From nobody Mon Mar 21 17:56:05 2022 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id A2830191398A for ; Mon, 21 Mar 2022 17:56:23 +0000 (UTC) (envelope-from prvs=07205606a=roger.pau@citrix.com) Received: from esa2.hc3370-68.iphmx.com (esa2.hc3370-68.iphmx.com [216.71.145.153]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KMj3V1htpz3PwV for ; Mon, 21 Mar 2022 17:56:22 +0000 (UTC) (envelope-from prvs=07205606a=roger.pau@citrix.com) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1647885382; h=date:from:to:cc:subject:message-id:references: content-transfer-encoding:in-reply-to:mime-version; bh=Vvbqj0utJCcEE0Q9fTPBlT9T5glwFbrnoAFxBG9r8YM=; b=R9uoealhQc65uTicTf3lRJIulgZ7vCa6zFBJngPI+uwDh4GmeFaKcz3R kMFJQLBlbuqlW7CiVpONyHQkjejqefKS9zRdS7uAAiHkViX1s91rVFr0N ELjw8WM+B0TPANWtw5vbjfkNMYYVKne390az4R8muo2hWtJdATzvn3cqn 8=; X-SBRS: 5.1 X-MesageID: 66796411 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.156.83 X-Policy: $RELAYED IronPort-Data: A9a23:Rn8er6MA8zS7TP7vrR2nl8FynXyQoLVcMsEvi/4bfWQNrUpw0DAGn GtJWGyCPKnYYGfxc992YIq1oE5V6MDWn9ExTwto+SlhQUwRpJueD7x1DKtR0wB+jCHnZBg6h ynLQoCYdKjYdleF+lH1dOKJQUBUjclkfJKlYAL/En03FFcMpBsJ00o5wbZi2Ncw27BVPivW0 T/Mi5yHULOa82Yc3lI8s8pvfzs24ZweEBtB1rAPTagjUG32zhH5P7pGTU2FFFPqQ5E8IwKPb 72rIIdVXI/u10xF5tuNyt4Xe6CRK1LYFVDmZnF+A8BOjvXez8CbP2lS2Pc0MC9qZzu1c99Zk Yx0rYOJSw4QNKTukrUAUiNnQjl+BPgTkFPHCSDXXc27ykTHdz3nwul0DVFwNoodkgp1KTgQr 7pCcmlLN03dwbLtqF64YrAEasALNs7kMZlZonh95TrYEewnUdbIRKCiCdpwgmlv154SRK+2i 8wxKh1hYijufiZ0B3Q9OZQjuMiJhUvDWmgNwL6SjfVuuDWCpOBr65DnLdrPfcOiQ8tflU+c4 GXc8AzRDAsXOdaExX+V/3SEiPXCkCngVMQVDrLQ3uxnjECL3X0IIBITXFq/5/K+jyaDt8l3c hJOvHB09O5rqRLtHoKVswCETGCs50clQ8VpEeQD4givwPvJ4jymV3AEd2sUADA5j/MeSTsv3 16PutrmAz1zrbGYIU6gGqeoQSCaYnZMczJbDcMQZU5cuoS4/tlv5v7aZow7eJNZmOEZDt0ZL 9qiiCElz4segscQv0lQ1QCW2mn8znQlo+Nc2+k2Yo5Hxl4iDGJGT9bxgbQ+0RqnBNzFJmRtR FBex6CjABkmVPlhbhClTuQXB62O7P2YKjDailMHN8B/q2v9oib6JNAJu2AWyKJV3iEsI2KBX aMukVkJuM870IWCM8ebnL5d++x1lPO9RLwJp9jfb8ZUY4gZSeN01HoGWKJk5Ei0yBJEufhmY f+zKJ/wZV5HWfUP5GfnHI81jO50rh3SMEuOHPgXOTz8iuHADJNUIJ9YWGazghcRt/rV/l+Kr 44Ba6NnCXx3CYXDX8UeyqZKRXgiJnknH5Hm7ctRc++IOA19H289TfTWxNscl0ZNxsy5Ss+gE qmBZ3Jl IronPort-HdrOrdr: A9a23:C/PRQKE5aKcAwBkSpLqE7MeALOsnbusQ8zAXPidKOHtom62j5q STdZEgviMc5wx8ZJhNo7+90cq7IU80l6Qa3WB5B97LNmTbUQCTTb1K3M/PxCDhBj271sM179 YET0GmMqySMbGtt7eZ3DWF X-IronPort-AV: E=Sophos;i="5.90,199,1643691600"; d="scan'208";a="66796411" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KpeEWqzoYZawnS4b+Gb/IL/dVEyL7Pd91Tmaa8YzwqgKJJnS/sHaiU9y57y92qKWcFAc0b5ZVakUgeRSK83Rr04Hy+y4SQ2++mzNlzjPUs0r6dKzs2em6Y0/jIT9kDLNlL/sUaphYPcss4erXsQEvpxkQIbJCIYHgLDNCWfcXOWJxlU/3+RLsq8yocgFhU0gPAZVYEeH6nd3o1BAkTUDJtVTuYWvBeFIS69InigK5+4/P6ioVCVwX+kD/lM1bj6gy9afr23q4EyI1q7qUTC3Xpy725HGu9LJfUjABQ/y+rnpwZC9vCdtv2WDQc2V6ttAE+ILUmvpn2CerKGC8jQY/w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=eNaGV7eV4QFfNUqMMN322Yd7UghgRTOsyx92BBIbais=; b=hvN7kWj0dw/3yrPnnwOc0lGP9+0ROiP84FSD6JGJj9/CwCRCdhQ5iLxfZFyMS0Hhv3lYq5WYIfQBTAWwE0ITdqH7jN/zPem5CRcQ0Hqfz/sxzcQzXkK0539Uo9hLNjE6u54MoVXT3MAcCQXaGKI3msKG7xkjL8omPz6E3laL4Dxf+iZY/GMgjSD348t2hgQLrRqjLcxo4iUW8YLGIjujS76Xqn0obDrD60IApR6vrAVpIfw5+T1npHxMCwFMtQW9iF1p6ZUkk52nnB2UuNp6zmWbGotA34tOb9qmSuGw21QGsxKXAgyqCruscSLkJTtXQB2wt4OKGJDmztjCb2vqLw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com; dkim=pass header.d=citrix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eNaGV7eV4QFfNUqMMN322Yd7UghgRTOsyx92BBIbais=; b=bWv6E6ZRRfjlP//VSLE/CojvwAC9+ToYQrlZYOjFf3JrUwYUuO8W0FrvLFpFQR73gz0sB+0vXmdG4uxDuGHJkIM17OnXNJ1mcdX6mGF/pwY+l5kwl79XTR56Rxt46Rsz7fSCmAlsZLzyhh66KgJeo+rc/B5CI29zDiX6oc2B2rc= Date: Mon, 21 Mar 2022 18:56:05 +0100 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Ze Dupsys CC: , Subject: Re: ZFS + FreeBSD XEN dom0 panic Message-ID: References: <5dfdecd5-f94d-29b4-791e-0adde5405cf5@gmail.com> Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-ClientProxiedBy: LNXP265CA0072.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:5d::36) To DS7PR03MB5608.namprd03.prod.outlook.com (2603:10b6:5:2c9::18) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 73143ce9-1482-4e53-3f59-08da0b64157a X-MS-TrafficTypeDiagnostic: DM6PR03MB4329:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: aeADm2dIfntiFsUrneofofr5EPxx3PY7Wd122zA8InlVi9bGF2vekvNAUIf6Dlk9JHzF8tEeEwHXHJwmxDziIyzC8vKnyyOPEAs+67KQmhvgXSXit3M2b7mQGF1JWz4si5P39IfZFcDnJriGQFNL5xiGxctBave1zMu+BV6cxExSZuXfHXfySCydj3FQO2KMX+TpdkR8Qz952fHEIkSuCSJLEa0KPQWEhmya8mPKodIbn6i4X1NOPwjefFSkXWS7+c11Aw2p/+jyppowpToIjCA0rN2FIAIqRsYwlMBdhJ/E4RfEYF6UNMu8lCSEGvlDyYCWouIXshsUOxQO31DPKg9vdiKPKLDfLqM/1kUfMw+Pn4PEQ9xox0oUZ8XWzD8E+J3GfHOi+16Nrb0aJbKqkcSPG6sqUWkDg9Jgkqns2+Dv1gwzdzotAlCj3E9zVjYs8jg/9s/pg4L7QoxNZZptHpI2F1fL9pX5l5SEK2h/Rd+G2XJUFY605yaBW75HfjqXKRnuG6C0ESgA+vHOrofxgaQlZF89CpNPNzspt52wvSfrd8zq4V8ngUz8O5ENU/vUTs+ZUdbbOdO+d2m4R3hGClwwrZAva+mhY0nEg8RE80asJXJv2MJONlpE7P4quVl0rdhvgOoaLVECXmTVz6Pzgw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(4636009)(7916004)(366004)(85182001)(186003)(26005)(38100700002)(9686003)(6512007)(8936002)(4326008)(66556008)(66476007)(66946007)(316002)(8676002)(83380400001)(6916009)(33716001)(508600001)(86362001)(6486002)(6506007)(2906002)(6666004)(82960400001)(5660300002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?YUgzWHVGS3NoSDZINmZ0dmt4d1BSYlZQK0FjV0lXTURreUhUdW1HV3Ard2Ra?= =?utf-8?B?MlY5amZ6aXdyOHp2dGVHaVlzUG41V0RraXJJbXVoc2lVOVR2Nzd5UEszUjNZ?= =?utf-8?B?U3BNeEVNVk5kZ3c4bkNkZkNxWlh1Q3cwQ1FUemx3MW9nTjI4WTlYVVBKbnhQ?= =?utf-8?B?UW5BaHpvVXFDZWN0RHZzdDBHdmJhQXlXdzExalpYQnFLRVBlUGdqNlFOYi9Y?= =?utf-8?B?dUhncXJXSWRqcHlCVEpRemk5RndJUXM1WW9XSkZSazBZU3ROUGJnMG80c09o?= =?utf-8?B?OVNpZVFJdlg0aUgvOHdwUWF2ZUszQVNSYnk3Mk91d0pjTmo4WldKaXVjcFdE?= =?utf-8?B?SlNDNDZmbXFnSGdmRDdMNDRSMWdiTUVuN0N5dU9KYituR2NxMzB4L0xhdGJp?= =?utf-8?B?eDc1WjFlSitMTGwrQ3pYSUJyazg2ZE5Pdjk2M3RrVWsxdlRYdCtpNnY5S2J2?= =?utf-8?B?SkpwMkR1ZFQ3UjlNTmc0aU4ybi9rUGpsSTVmc1pUQVNXOWQ5K3lyZzFuVjdZ?= =?utf-8?B?V00xdGIraVJyNW0xVDlRMm1JM05sdjZTNDlQZno5OFR0RElweEc1MEhBalZv?= =?utf-8?B?dU5ZVzdyYkkyZWV4ZzNaVXhBVkgzRS9NcXRCeGV4MnlYNDcyQUUyNEk3RzA0?= =?utf-8?B?RXM3ZFIvakY1MnVEK1c3V3lqVjdnS09NdUxKTzMzc2o1TG0rMFI0SThraEJx?= =?utf-8?B?OG5PYVJZS3grZmswUEZrSGFoeEI3MXpIQzJWY1ZRZE02WmpLNFR1RTZJYWVy?= =?utf-8?B?R1E2QlNEM3hYZ1d3QkVpMVVGUE5XT2E5ZWpEalNoZ2Mrak1jZlZBc0ZReVdO?= =?utf-8?B?V1kvSmNSNzVtcVlMR3hKSWhMWGtwRU9xSEtaejJPbEtkUFZjYThQTWhuUzRw?= =?utf-8?B?emVtaitVYUVSN2dPVll0NlV1TTF3dGsvZ0NlNWJQRmlkZDBZb2tUWWJPL0lP?= =?utf-8?B?NHp2bUNlYWsvaHAxZlVKajdvQXBoY2pHb0poM2Nsd2ZDbHZ5anhjSVJQRkFO?= =?utf-8?B?dWhTVklZcW11VVhMeUwzSDEreUM2SmJ6UVh4MVEwTUEyeUxHUVpZcHBjZHR0?= =?utf-8?B?N1FjZXR2UEFsV3k5K0ZwREFSMWg1dTl3dUJLNVRpOVM0Zzh2MGlKNEZKQmVD?= =?utf-8?B?cVJUY0ljTWd2dk5JalBpbFhKTDQxOE95Ulg5cGVEWTVxSmJlN3pzMDVxL2Y5?= =?utf-8?B?YzFSREZFZDlQb0VoclBvU2ErQlFsSW5GNTluN3k2MzlrQzNGQ3BBWEltcnNL?= =?utf-8?B?SHRaTENqQm1qR3FGNHhyeVAvSWFCMFVIcEpVSkFaL0c4TEVrZ2N4dFZSRmxH?= =?utf-8?B?K3BKTFZmNkN4MDA5dkJHdDFJM2F3djF6OHZCVVA2Z3M4MmtTdGg3SXhuNFRJ?= =?utf-8?B?Si82ZWVIMU5VNVEyazh3ZnV3TGtaYk1FMFZnTFZtYUVDNVVGbnpzWWFrL0M5?= =?utf-8?B?aWhyTGtwRXREbW1HanNaNkJIMTArY2ZubnViTkRLNjF3QmxzTzV0YTVSd21D?= =?utf-8?B?UGY5aC8yaWtlT284VVE3VDArY25rc3lyOEZZUlE2aXpZT3Z6WlVYdGpOdmNa?= =?utf-8?B?Z2FzcEZuU1g2ZmRZSDB5VW40WG9ONytlZEREaGFiZWFoRnEwcWdZUXdsREZZ?= =?utf-8?B?K05YbHdkWjZrbXdpMDVMc0NWMEVBVlY5UVFxVlNMVG9XenJ3OFRBRjlkaE1V?= =?utf-8?B?QTJzOHJRMXd6RFhZQlM3dkJQUXZZbTZzMFpLK01CbkhJYnE2TlpGaXlIM3N1?= =?utf-8?B?NWZzeXRnaE9uc2pjZ1dabUhLYndEUTNxK1BhU0tYVTY0OEd3bFpOYThTZVpa?= =?utf-8?B?Tjk5ZFc5VWdmTVdaTmhCNDlJM1JmeGxuQ3ZEQm5ZekRBV25GVGtqcU9Lb3Iy?= =?utf-8?B?Wm5QVUl1aTdkVjNuY2hKS2tWWS9EOUdTUWpzdXNoRUFMTkw5RmM3cWhoZVNR?= =?utf-8?B?NklDbUViK01JbU52ampNUDlIMHNPTjhzYWxRTlhlQlhhTnpiYjgvNktaZlRy?= =?utf-8?B?cVpMd3pFU05BPT0=?= X-MS-Exchange-CrossTenant-Network-Message-Id: 73143ce9-1482-4e53-3f59-08da0b64157a X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Mar 2022 17:56:11.3658 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: qbD8bEAgU9jDPalTEgV1OzTrXxoWrPf6PhRmbspivM7wtpPw3NG0eKL5siLCmDXmSDJlBh7MEENIODc96oAAxQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR03MB4329 X-OriginatorOrg: citrix.com X-Rspamd-Queue-Id: 4KMj3V1htpz3PwV X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=citrix.com header.s=securemail header.b=R9uoealh; dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=bWv6E6ZR; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=reject) header.from=citrix.com; spf=pass (mx1.freebsd.org: domain of "prvs=07205606a=roger.pau@citrix.com" designates 216.71.145.153 as permitted sender) smtp.mailfrom="prvs=07205606a=roger.pau@citrix.com" X-Spamd-Result: default: False [-5.64 / 15.00]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+exists:216.71.145.153.spf.hc3370-68.iphmx.com]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+]; DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject]; RCVD_IN_DNSWL_MED(-0.20)[216.71.145.153:from]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_COUNT_ZERO(0.00)[0]; MIME_TRACE(0.00)[0:+]; R_MIXED_CHARSET(0.56)[subject]; ASN(0.00)[asn:16417, ipnet:216.71.145.0/24, country:US]; FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=07205606a=roger.pau@citrix.com]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; FORGED_SENDER_VERP_SRS(0.00)[]; DWL_DNSWL_LOW(-1.00)[citrix.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[freebsd-xen]; MID_RHS_NOT_FQDN(0.50)[] X-ThisMailContainsUnwantedMimeParts: N On Mon, Mar 21, 2022 at 05:35:15PM +0100, Roger Pau Monné wrote: > On Mon, Mar 21, 2022 at 04:07:48PM +0200, Ze Dupsys wrote: > > On 2022.03.21. 13:14, Roger Pau Monné wrote: > > > I think the problem is not likely with the xenstore implementation > > > (ie: xs_talkv) but rather a race with how the FreeBSD kernel detects > > > and manages addition and removal of devices that hang off xenbus. > > > > > > I'm afraid there's too much data below for me to parse it. > > > > Understood. Sounds more tricky than i thought. What could i do to make data > > more useful? > > I have another patch for you to try. This will make the system a bit > chatty, let's see what you get. Forgot to mention: when testing the patch attached to the previous email there's no need to push the system until you get a panic. Just detecting when you have stale xbbd entries in sysctl would be enough, or alternatively when you start to see entries in the output of `xenstore-ls -fp` like: /local/domain/0/backend/vbd/XX/XXXXX = "" (n0) /local/domain/0/backend/vbd/XX/XXXXX/feature-barrier = "1" (n0) /local/domain/0/backend/vbd/XX/XXXXX/feature-flush-cache = "1" (n0) /local/domain/0/backend/vbd/XX/XXXXX/max-ring-page-order = "5" (n0) Note the lack of a '/local/domain/0/backend/vbd/XX/XXXXX/state' node. At that point I would request that you attach the output of `xenstore-ls -fp` together with the full serial log since the system booted. You might not need a lot of iterations to trigger that state. Thanks, Roger.