From nobody Mon Mar 14 08:06:58 2022 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id EDA491A1702D for ; Mon, 14 Mar 2022 08:07:01 +0000 (UTC) (envelope-from zedupsys@gmail.com) Received: from mail-lf1-x132.google.com (mail-lf1-x132.google.com [IPv6:2a00:1450:4864:20::132]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KH8Jj1QxPz3pgG for ; Mon, 14 Mar 2022 08:07:01 +0000 (UTC) (envelope-from zedupsys@gmail.com) Received: by mail-lf1-x132.google.com with SMTP id 3so25598614lfr.7 for ; Mon, 14 Mar 2022 01:07:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=subject:to:cc:references:reply-to:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding:content-language; bh=4xxOeJhLHt/sBb+3EHK/jdHbGdkBjUC3Yn6s7KNLgcI=; b=hD9T9Fxrfo/JJb+/GAvYk7dfnZBp+A5hi6zSC60NQyBlImYR9FD+kbD+NP9F6ohQ2e B581oh7R+GVPpYltZXeZWpXPrpXszwJX0FooSUBwc/acKMcVz+YgGsg5ATx4LJQcMw6q nrwHDu7oq/0ePcdabUIWeE5ljcfupeweb+48GqhQObGwO1RTUeYnmX+N0rBlSNTLXWCq 1QwLc2JzK2J629llGfOv0DeGTrTWH1j3AuqeHinbast97mQ5C+UlNtVWZ8w6YWex0jmk ZDd5cVEZfiQNAkRCEdrPNRNC1F6Iz/qfvZEHN/6Z/gD1NJfbweD4K02+f4q3jlD/YcrF HZXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:subject:to:cc:references:reply-to:from :message-id:date:user-agent:mime-version:in-reply-to :content-transfer-encoding:content-language; bh=4xxOeJhLHt/sBb+3EHK/jdHbGdkBjUC3Yn6s7KNLgcI=; b=ITTzCZnwG6QsLGuPahN6L+eXOwhBss8FAi8nubTrYdlA0xMX+EOT+0L6r0iQE7PJ32 CbiyUWG32odCwz6rssfTSSXl2lvfGfCaN3it1eYm5831QEGEK7RMbRYSCQU447W2ykyb pbj5S2RcqMLqkr5kVZ4AdJJsi35Ea/ac15zHRr/ZqZUceWNXlQrLnk6ubOJ7F+LTz+rI n+Ek6juN6bcBSMypNVeSuQXL7F3jUGAsOxVbD9KxDgNbWbw1EyrAs0MBQh6GKrtk4/ai xYXuY1bq46TSWgQYilh4WhSk018gfcxn4lwJZdMoSPuk5ij4mD9r3WOCYqzj29Ur0zU1 7dvQ== X-Gm-Message-State: AOAM530khHMP8h5VZ3vmh7J9sYNtmk3AZ63KbwiF1+2KNJR5yYY7Nh9u n1EEZpBZZpPobBIFpXAdB5px135i/D4= X-Google-Smtp-Source: ABdhPJwwJeZZUiBGl3b8+PaX//P8LKSpHd0VFiTg1jMeZAg4dyBme39C5R4HBh/SECfqUvfbrdoCGQ== X-Received: by 2002:a05:6512:b19:b0:446:f1c6:81bd with SMTP id w25-20020a0565120b1900b00446f1c681bdmr13031381lfu.320.1647245213655; Mon, 14 Mar 2022 01:06:53 -0700 (PDT) Received: from [10.3.0.1] ([213.110.65.3]) by smtp.googlemail.com with ESMTPSA id v1-20020a2e6101000000b002480c4f33d5sm3160645ljb.132.2022.03.14.01.06.52 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 14 Mar 2022 01:06:53 -0700 (PDT) Subject: Re: ZFS + FreeBSD XEN dom0 panic To: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= Cc: freebsd-xen@freebsd.org, buhrow@nfbcal.org References: <202203011540.221FeR4f028103@nfbcal.org> <3d4691a7-c4b3-1c91-9eaa-7af071561bb6@gmail.com> <5dfdecd5-f94d-29b4-791e-0adde5405cf5@gmail.com> Reply-To: zedupsys@gmail.com From: Ze Dupsys Message-ID: Date: Mon, 14 Mar 2022 10:06:58 +0200 User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.12.0 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Rspamd-Queue-Id: 4KH8Jj1QxPz3pgG X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=hD9T9Fxr; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of zedupsys@gmail.com designates 2a00:1450:4864:20::132 as permitted sender) smtp.mailfrom=zedupsys@gmail.com X-Spamd-Result: default: False [-0.62 / 15.00]; HAS_REPLYTO(0.00)[zedupsys@gmail.com]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; REPLYTO_ADDR_EQ_FROM(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.15)[-0.146]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; MIME_GOOD(-0.10)[text/plain]; FREEMAIL_REPLYTO(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org]; NEURAL_SPAM_MEDIUM(0.28)[0.281]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.24)[0.245]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::132:from]; MLMMJ_DEST(0.00)[freebsd-xen]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N I'd like to share more analysis on given problem. I do not know if this somehow helps or not, but i have noticed that across all my saved serial log outputs, panic messages follow after some of these lines. .. (XEN) HVM d34v0 save: TSC_ADJUST (XEN) HVM d34v0 save: CPU_MSR (XEN) HVM34 restore: CPU 0 xnb(xnb_detach:1330): xnb(xnb_detach:1339): .. => panic Most of panics are like this .. (XEN) HVM d26v0 save: TSC_ADJUST (XEN) HVM d26v0 save: CPU_MSR (XEN) HVM26 restore: CPU 0 .. => panic .. (XEN) HVM d42v0 save: TSC_ADJUST (XEN) HVM d42v0 save: CPU_MSR (XEN) HVM42 restore: CPU 0 xnb(xnb_detach:1330): xnb(xnb_detach:1339): xnb(xnb_detach:1330): xnb(xnb_detach:1339): .. => panic This one i think had different stressing conditions than other's, but i don't remember .. (XEN) HVM d660v0 save: CPU_MSR (XEN) HVM660 restore: CPU 0 (XEN) d659v0: upcall vector 93 spin lock 0xffffffff81eaa780 (sched lock 1) held by 0xfffff8020152d000 (tid 100434) too long timeout stopping cpus panic: spin lock held too long .. => panic For serial output in middle when there are no crashes i have noticed that there are at least 2 different execution paths. For most VM's boot flow continues with serial lines like these: .. (XEN) HVM1 restore: CPU 0 xnb(xnb_probe:1123): Claiming device 0, xnb xnb(xnb_attach:1267): Attaching to backend/vif/1/0 xnb(xnb_frontend_changed:1391): frontend_state=Initialising, xnb_state=InitWait (d1) HVM Loader .. For some though, there are lines like these, but they still boot, it just seemed that these lines might be a possible continuation for "unsuccessful panic". .. (XEN) HVM3 restore: CPU 0 xnb(xnb_detach:1330): xnb(xnb_detach:1339): xnb(xnb_detach:1330): xnb(xnb_detach:1339): xnb(xnb_probe:1123): Claiming device 0, xnb xnb(xnb_attach:1267): Attaching to backend/vif/3/0 xnb(xnb_frontend_changed:1391): frontend_state=Initialising, xnb_state=InitWait (d3) HVM Loader .. Why those lines starting "xnb(xnb_detach:1330):" do not have any message? Could it be that there is a bad pointer to message buffer that can not be printed? And then sometimes panic happens because access goes out of allowed memory region? Line numbers are just somewhat informational, since those messages are from across all my tests, various configs, versions. Yesterday i set up system with FreeBSD 13.1-STABLE, still can crash, same panic. What i do not know about those xnb messages is, to which VM they are related, since serial output is shared and on parallel while VM1 might be created, VM2 is starting or being destroyed. Thanks.