From nobody Thu Mar 10 11:41:28 2022 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 99FE219FE88A for ; Thu, 10 Mar 2022 11:41:13 +0000 (UTC) (envelope-from zedupsys@gmail.com) Received: from mail-lj1-x22a.google.com (mail-lj1-x22a.google.com [IPv6:2a00:1450:4864:20::22a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KDnFh5DTKz4x8X for ; Thu, 10 Mar 2022 11:41:12 +0000 (UTC) (envelope-from zedupsys@gmail.com) Received: by mail-lj1-x22a.google.com with SMTP id s25so7301169lji.5 for ; Thu, 10 Mar 2022 03:41:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=subject:to:cc:references:reply-to:from:message-id:date:user-agent :mime-version:in-reply-to:content-language; bh=7Z+LMTzU4xxXWXI7RKfOZPZgbubePPV/duqXQCATFSQ=; b=Tdm2gRddN1cVCASalTQH0dy0eurwskwkw++QvQyzDTA5enKGONQ3+7e6txMgzezaJn w/7gCOkzW2jh4/a1K6UIgZKHIuAvJnaAH2lvytZq0QNS91SFh84QZM6b8aVupmWOPqQO MgoD0Ms+7wPEyibE8g+No1p1ubebV/YOr5D6HCjm5sJz1JD85A7lNVpaUnpfiZanODgk 5WgF5TgFrCBZ/lZThvitxidLWzVVzgVlcgPKTOWebg6O3xa5eXSr5JRy6rHGioHBzPvJ XY1yK6G137b9FEn76A+c2im6wdxOzM9j7vtk+fk0WRlFtp0d6aX8lBQXMvlDlT75x0bW DseQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:subject:to:cc:references:reply-to:from :message-id:date:user-agent:mime-version:in-reply-to :content-language; bh=7Z+LMTzU4xxXWXI7RKfOZPZgbubePPV/duqXQCATFSQ=; b=dj5UVo5BdricfmP63jLlyL8Ir0VTDybBtp67P3W9SO9i+6WulGFVVvQ/w700LJN979 eNDUx2s3kHxc++4kWYxMuZxN0e3xn0Jpe+Iygu5JfslO3uidUZVOem1BhHETjvjWyV1n aR4OsXOCe9m8yv+gcrKqBDNhdhBplPQGbs7T5sSMkQyfImn/5Awluui1oS2u4HDpYNh+ hjGr3zeobLe3afw8QyoPFTs9Jkz2hnKFllEOWfNwrnCCQ1mI28MZXrUhUhnc8GeGz4jc NBlN70Oh92upsKm+TdjlpKDNgpoJQ2PuSM4YVoYSe06yysjqkE2PEnUcv6h+aF1qcY+j dNCg== X-Gm-Message-State: AOAM530TNOy1hGg8tZ8zDCh8eCvK837n14BI4ViUBBmcYrEnqx8FcXVh ebpWSu0blaTxkcLWkzXOo0c= X-Google-Smtp-Source: ABdhPJz2vgOutmVglwHJPE1dxgi8s9yvWt5blfes6pwJH/qVQjmtVIUfJOq8WUuWoCIBCxVKHqUU4A== X-Received: by 2002:a2e:824b:0:b0:246:1246:d830 with SMTP id j11-20020a2e824b000000b002461246d830mr2763629ljh.267.1646912471311; Thu, 10 Mar 2022 03:41:11 -0800 (PST) Received: from [10.3.0.1] ([213.110.65.3]) by smtp.googlemail.com with ESMTPSA id 25-20020ac25f19000000b0044858b3cc20sm721624lfq.40.2022.03.10.03.41.10 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 10 Mar 2022 03:41:10 -0800 (PST) Subject: Re: ZFS + FreeBSD XEN dom0 panic To: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= Cc: freebsd-xen@freebsd.org, buhrow@nfbcal.org References: <202203011540.221FeR4f028103@nfbcal.org> <3d4691a7-c4b3-1c91-9eaa-7af071561bb6@gmail.com> Reply-To: zedupsys@gmail.com From: Ze Dupsys Message-ID: <5dfdecd5-f94d-29b4-791e-0adde5405cf5@gmail.com> Date: Thu, 10 Mar 2022 13:41:28 +0200 User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.12.0 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/alternative; boundary="------------0001BD4272D7E3B6557D5819" Content-Language: lv X-Rspamd-Queue-Id: 4KDnFh5DTKz4x8X X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=Tdm2gRdd; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of zedupsys@gmail.com designates 2a00:1450:4864:20::22a as permitted sender) smtp.mailfrom=zedupsys@gmail.com X-Spamd-Result: default: False [-0.97 / 15.00]; HAS_REPLYTO(0.00)[zedupsys@gmail.com]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; REPLYTO_ADDR_EQ_FROM(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_SPAM_SHORT(0.96)[0.957]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; FREEMAIL_REPLYTO(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org]; NEURAL_SPAM_MEDIUM(0.07)[0.074]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::22a:from]; MLMMJ_DEST(0.00)[freebsd-xen]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N This is a multi-part message in MIME format. --------------0001BD4272D7E3B6557D5819 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit On 2022.03.09. 10:42, Roger Pau Monné wrote: > On Sun, Mar 06, 2022 at 02:41:17PM +0200, Ze Dupsys wrote: >> Then i caught another sysctl variable that is growing due to XEN, >> "kern.msgbuf: Contents of kernel message buffer". I do not know how this >> variable grows or by which component it is managed, but in VM start/stop >> case it grows and contains lines with pattern like so: >> .. >> xnb(xnb_rxpkt2rsp:2059): Got error -1 for hypervisor gnttab_copy status >> xnb(xnb_ring2pkt:1526): Unknown extra info type 255.  Discarding packet >> xnb(xnb_dump_txreq:299): netif_tx_request index =0 >> xnb(xnb_dump_txreq:300): netif_tx_request.gref  =0 >> xnb(xnb_dump_txreq:301): netif_tx_request.offset=0 >> xnb(xnb_dump_txreq:302): netif_tx_request.flags =8 >> xnb(xnb_dump_txreq:303): netif_tx_request.id    =69 >> xnb(xnb_dump_txreq:304): netif_tx_request.size  =1000 >> xnb(xnb_dump_txreq:299): netif_tx_request index =1 >> xnb(xnb_dump_txreq:300): netif_tx_request.gref  =255 >> xnb(xnb_dump_txreq:301): netif_tx_request.offset=0 >> xnb(xnb_dump_txreq:302): netif_tx_request.flags =0 >> xnb(xnb_dump_txreq:303): netif_tx_request.id    =0 >> xnb(xnb_dump_txreq:304): netif_tx_request.size  =0 >> .. >> >> Those lines in that variable just keep growing and growing, it is not that >> they are flushed, trimmed or anything. Each time i get the same message on >> serial output, it has one more section of error appended to "same-previous" >> serial output message and sysctl variable as well. Thus at some point serial >> output and sysctl contains a large block of those errors while VM is >> starting. So at some point the value of this sysctl could be reaching max >> allowed/available and this makes the system panic.  I do not know the reason >> for those errors, but actually if there was a patch to suppress them, this >> could be "solved". Another diff chunk might be related to this: >> +dev.xnb.1.xenstore_peer_path: /local/domain/7/device/vif/0 >> +dev.xnb.1.xenbus_peer_domid: 7 >> +dev.xnb.1.xenbus_connection_state: InitWait >> +dev.xnb.1.xenbus_dev_type: vif >> +dev.xnb.1.xenstore_path: backend/vif/7/0 >> +dev.xnb.1.dump_rings: >> +dev.xnb.1.unit_test_results: xnb_rxpkt2rsp_empty:1765 Assertion Error: >> nr_reqs == 0 >> +xnb_rxpkt2rsp_empty:1767 Assertion Error: memcmp(&rxb_backup, >> &xnb_unit_pvt.rxb, sizeof(rxb_backup)) == 0 >> +xnb_rxpkt2rsp_empty:1769 Assertion Error: memcmp(&rxs_backup, >> xnb_unit_pvt.rxs, sizeof(rxs_backup)) == 0 >> +52 Tests Passed >> +1 Tests FAILED > So you have failed tests for netback. Maybe the issue is with > netback rather than blkback. Just found that there are no errors with network, most probably. Based on https://reviews.freebsd.org/D9234?id=24172#201024, since i was monitoring sysctl variables with 'sysctl -a', most probably that did "sysctl dev.xnb.0.unit_test_results" implicitly for each xnb interface, thus the extra output, since reading value actually runs tests. So now i upgraded my monitoring command to 'sysctl -a -N'. Never did i expect that sysctl variable reading could trigger test suite execution. I did not understand from comments though - how do you instantiate xnb interface? "ifconfig xnb create" returned: ifconfig: SIOCIFCREATE2: Invalid argument Thanks. --------------0001BD4272D7E3B6557D5819 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit On 2022.03.09. 10:42, Roger Pau Monné wrote:
On Sun, Mar 06, 2022 at 02:41:17PM +0200, Ze Dupsys wrote:

Then i caught another sysctl variable that is growing due to XEN,
"kern.msgbuf: Contents of kernel message buffer". I do not know how this
variable grows or by which component it is managed, but in VM start/stop
case it grows and contains lines with pattern like so:
..
xnb(xnb_rxpkt2rsp:2059): Got error -1 for hypervisor gnttab_copy status
xnb(xnb_ring2pkt:1526): Unknown extra info type 255.  Discarding packet
xnb(xnb_dump_txreq:299): netif_tx_request index =0
xnb(xnb_dump_txreq:300): netif_tx_request.gref  =0
xnb(xnb_dump_txreq:301): netif_tx_request.offset=0
xnb(xnb_dump_txreq:302): netif_tx_request.flags =8
xnb(xnb_dump_txreq:303): netif_tx_request.id    =69
xnb(xnb_dump_txreq:304): netif_tx_request.size  =1000
xnb(xnb_dump_txreq:299): netif_tx_request index =1
xnb(xnb_dump_txreq:300): netif_tx_request.gref  =255
xnb(xnb_dump_txreq:301): netif_tx_request.offset=0
xnb(xnb_dump_txreq:302): netif_tx_request.flags =0
xnb(xnb_dump_txreq:303): netif_tx_request.id    =0
xnb(xnb_dump_txreq:304): netif_tx_request.size  =0
..

Those lines in that variable just keep growing and growing, it is not that
they are flushed, trimmed or anything. Each time i get the same message on
serial output, it has one more section of error appended to "same-previous"
serial output message and sysctl variable as well. Thus at some point serial
output and sysctl contains a large block of those errors while VM is
starting. So at some point the value of this sysctl could be reaching max
allowed/available and this makes the system panic.  I do not know the reason
for those errors, but actually if there was a patch to suppress them, this
could be "solved". Another diff chunk might be related to this:
+dev.xnb.1.xenstore_peer_path: /local/domain/7/device/vif/0
+dev.xnb.1.xenbus_peer_domid: 7
+dev.xnb.1.xenbus_connection_state: InitWait
+dev.xnb.1.xenbus_dev_type: vif
+dev.xnb.1.xenstore_path: backend/vif/7/0
+dev.xnb.1.dump_rings:
+dev.xnb.1.unit_test_results: xnb_rxpkt2rsp_empty:1765 Assertion Error:
nr_reqs == 0
+xnb_rxpkt2rsp_empty:1767 Assertion Error: memcmp(&rxb_backup,
&xnb_unit_pvt.rxb, sizeof(rxb_backup)) == 0
+xnb_rxpkt2rsp_empty:1769 Assertion Error: memcmp(&rxs_backup,
xnb_unit_pvt.rxs, sizeof(rxs_backup)) == 0
+52 Tests Passed
+1 Tests FAILED

So you have failed tests for netback. Maybe the issue is with
netback rather than blkback.

Just found that there are no errors with network, most probably. Based on https://reviews.freebsd.org/D9234?id=24172#201024, since i was monitoring sysctl variables with 'sysctl -a', most probably that did "sysctl dev.xnb.0.unit_test_results" implicitly for each xnb interface, thus the extra output, since reading value actually runs tests. So now i upgraded my monitoring command to 'sysctl -a -N'. Never did i expect that sysctl variable reading could trigger test suite execution.

I did not understand from comments though - how do you instantiate xnb interface?

"ifconfig xnb create" returned:
ifconfig: SIOCIFCREATE2: Invalid argument

Thanks.



  


--------------0001BD4272D7E3B6557D5819--