From nobody Thu Apr 14 07:49:27 2022 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4AB4E1B43CCE for ; Thu, 14 Apr 2022 07:49:42 +0000 (UTC) (envelope-from prvs=09619e58f=roger.pau@citrix.com) Received: from esa1.hc3370-68.iphmx.com (esa1.hc3370-68.iphmx.com [216.71.145.142]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KfBSP2Bw1z4tc3 for ; Thu, 14 Apr 2022 07:49:41 +0000 (UTC) (envelope-from prvs=09619e58f=roger.pau@citrix.com) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1649922581; h=date:from:to:cc:subject:message-id:references: content-transfer-encoding:in-reply-to:mime-version; bh=kThIC6pGNn0YEhnmJnsHd/o31FIaLOKAYDBbQzxvDSk=; b=fcbV/ENYneheODrrrLcM9OnqrXHO4jhULwnq4bFCBKxOKY36zOxPkLRj gn5KTn2ixE0YZc7+7gPkkq5LRFdI1mcFNGR7I4Y6OHHQbq6PB6t6kMnpc SZu5BhKdIeWCWh/IduTJmgyPabLcVKWQL/nxuN4W0soX4Hq5GjcA6sZE5 E=; X-SBRS: 5.1 X-MesageID: 69336493 X-Ironport-Server: esa1.hc3370-68.iphmx.com X-Remote-IP: 162.221.156.83 X-Policy: $RELAYED IronPort-Data: A9a23:cBOMM6tudQYXpC9vrG4OVUGJDefnVEBeMUV32f8akzHdYApBsoF/q tZmKWzQM/bYa2r8LdAgPNm38R8CsZHRx4RiGVQ4pC88Ei0W+JbJXdiXEBz9bniYRiHhoOOLz Cm8hv3odp1coqr0/0/1WlTZhSAgk/nOHNIQMcacUsxLbVYMpBwJ1FQyw4bVvqYy2YLjW1/U5 YuryyHiEATNNwBcYzp8B52r8HuDjNyq0N/PlgVjDRzjlAa2e0g9VPrzF4noR5fLatA88tqBb /TC1NmEElbxpH/BPD8HfoHTKSXmSpaKVeSHZ+E/t6KK2nCurQRquko32WZ1he66RFxlkvgoo Oihu6BcRi8ZBLLVnuciDSJ6MDwhD6NKorLXIWGG5Jn7I03uKxMAwt1rBUAye4YZ5vx2ESdF8 vlwxDIlN07ZwbjsmfTiF7cq1p9LwMrDZevzvll6yj7UF7A+SI3rSKTW/95Imjw3g6iiGN6AN pFCN2Y+N3wsZTVCFFIRBINhoN3w2FKgUDx7+AK6hPEotj27IAtZj+G2bYu9lsaxbc9Inlyfu UrA8W7yCxVcM8aQoRKL6H+hjPTD2DP2XqocDrC1/OVmxlqJyQQ7GhAaSEOgsOKRhUu3WtYZI EsRkhfCtoBrqhbtFIOkGUTl/jjU5XbwRua8DcUK0h6UzKnrvT27JXVbfDBNMtE2tNYPEGlCO kCyo/vlAjlmsbuwQH2b96uJoT7aBRX5PVPudgdfE1JbvoCLTJUby0uWE409SPLdYsjdQ2mY/ tyckMQpa1z/Z+Yv3r7zw13IiinESnPhHl9svVW/so5IA2pEiG+Zi26AtACzARVodt/xory9U J4swZn2AAcmV8/lqcB1aL9RdIxFHt7cWNEmvXZhHoM66xOm8GO5cIZb7VlWfRk1Y5ZeIWa3P BSI5Gu9AaO/2lPwPMebhKrrVawXIVXIT4y5Bpg4kPITCnSOSON31H43PhPBt4wcuEMtjbs+K f+mnTWEVh4n5VBc5GPuHY81iOZzrghnnD+7bc2rnnyPjOvFDFbIGOhtDbd7Rr1ghE9yiF6Oq Ig32grj40g3bdASlQGOqNBIfAFWdyVjbX00wuQOHtO+zsNdMDhJI9fawK87epwjmKJQl+zS+ Wq6VFMew1367UAr4y3TApy/QNsDhapCkE8= IronPort-HdrOrdr: A9a23:ND6Asa4B9BRsKsopcQPXwPTXdLJyesId70hD6qkXc20wTiX4rb HKoB11726WtN98YhAdcLO7VpVoI0mskqKdiLN5Vd3OYOCPghrPEGgI1/qB/9SPIU3DH8dmpN 5dm/cUMrbN5CJB4voSSjPIa+rIu+PqzJyV X-IronPort-AV: E=Sophos;i="5.90,259,1643691600"; d="scan'208";a="69336493" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=oXoe6NXKAkmPzxLaRIFWoAn9Je2pBEFLMErTZUVi4+Z9jfggya/qRPLPGb5aAx5UNQw3P3j6R53NHIqN/BLtrhQ2UEW2dsHv2ZbxlGtmQOq8BoLIPK895kHS4rGGsyxAntZ3lp+3RfP6A3HXekFPmjSSaJLks1It6wq5CSQ+lCVBaNe9eLmTlHhTUtw65tvETPdFgpErjqd330AxZNjgp/4oYtk8h1LUuOm1uFSHSl4kMoOOUq14oXtblj/tCypqEevrk5hzPD24IrMpL04W/bhoyaszw0DWdv3tmaVdh0DPpJEHTeSFPysl0dOI/BixgIR2lX2q0jA1KWVdfcmPMg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=2rq1r/Sfs+8UVVf/JV0u3TqTHggjrko+aiB19/kSe5s=; b=bPpI+KyTCvMDBwxHIQQJ3fIft8yHglBCZScdSrwXxYU5UBlj9B4hVDa/oc0jhgDvkN/ibHdk61wem9FQDPa/oBDM7Khv6k0SUaafNPN1ZBakGKdV+1J/oZ6TXuEXkhwJK6Ez7F2iOsSL3ckgRhxEbk0sOwi/1B2HvHsxk2oNrMIQfSt+5Ynd7RejxpV+NtfZkU2J1s+wp9nvS7/KB+cTHtT7DykzXc4I7ai2itsu5L/sbxMy2cy87lrWi30IReidEacVKuzYgBrOldC0qATvBLjwx8eyXIz8WSQJAgnxcif+FL9dGQohF4Y1mFHQBNAsyLi1r4UAJYG1CibFhoxGlg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com; dkim=pass header.d=citrix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2rq1r/Sfs+8UVVf/JV0u3TqTHggjrko+aiB19/kSe5s=; b=fjOZxa2pKYUWXKKB79IFBWAqaLzK9TfyxNX64AXDrjt6GzG9/WxcvGzW0wFZcQOgWoY0/5Qkjx4BF9cMxta95QMWpBew4fLwm7HXzKWN5z/Z38xDG/Q+dYsOAkZCvFoHcO8B7T988iL0BdkZDACJkbWTPPtmgvFQbYxWRWnCG20= Date: Thu, 14 Apr 2022 09:49:27 +0200 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Ze Dupsys CC: , Subject: Re: ZFS + FreeBSD XEN dom0 panic Message-ID: References: <639f7ce0-8a07-884c-c1cf-8257b9f3d9e8@gmail.com> <4da2302b-0745-ea1d-c868-5a8a5fc66b18@gmail.com> <48b74c39-abb3-0a3e-91a8-b5ab1e1223ce@gmail.com> <22643831-70d3-5a3e-f973-fb80957e80dc@gmail.com> <2dbf24f9-2bc8-a8d5-e31f-90ec2c4b64c5@gmail.com> Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <2dbf24f9-2bc8-a8d5-e31f-90ec2c4b64c5@gmail.com> X-ClientProxiedBy: MR1P264CA0051.FRAP264.PROD.OUTLOOK.COM (2603:10a6:501:3e::26) To DS7PR03MB5608.namprd03.prod.outlook.com (2603:10b6:5:2c9::18) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 5552b399-a954-4010-7b4b-08da1deb4fd2 X-MS-TrafficTypeDiagnostic: DM6PR03MB3948:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: /gpBkL2ag9ctEeUtAwkXECpHeBA33YqYmZMXjXuvTGrft/ANSPT8G7dTNn69AkwGXgrAUldpt0rY+nk9ZljIBUk97YLLi5OzERz56XEjVqHfOljNVh8NR/lwdAREUB0hyS3Xwg8phtc5rJr4XdJ4Ifs66l8oDejXDBwy5eVmJxrL5H8U89XEZFBhbJbp73maEza25reum6y+T9WaD0KL91dgnHBmM/McIx40DdOIte5Ymyn0bqRgoaMUNYJJVxF4n6HUfJO58zpE4/reQj171Re7G3vX273y4ANrUMENT37ZwETjCr2v+Yy1NJS3JD34IRDskhjqfS7BtZHa5+wzGRokrTettjxw8j83VW4U7B+OXP53QT/OJZRaPYYZNG8ydIiG/e57kx0X9Y5FkgoRKJTvLRKZ85o2P058M5TXGlm8Fkfs66cLnw99eQME08hLcW96tN4wvB8Urqiv+THTdEnOb6Ob5OC3YGpMLBCwfAUwX8yROMR6IM0Tg6p1XNwtZ3uSaMzggh8lsSccYxiaa73ywC0+yN4KgC+Z751bUCYSQ6S4BPc+5tnTcabFK6XsNBRHUoAKnUMSRhpAyntcjJjuRrmI2ZPkNs9Oy1eevSzP3k3l/eIR+gpTJTNCK4aVCVJP1EmQS9BU8yqXlzwdmps4gbMZJeNJUDdE7ew4sOkH/GM4Cp8RZnyfWfWMFwzAFu5E+8okwp/9/yDFlFyxv0xBGIo3zp9+ejBg0ydNwSo= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(7916004)(4636009)(366004)(6666004)(83380400001)(33716001)(8936002)(66556008)(6916009)(85182001)(2906002)(9686003)(6512007)(6506007)(38100700002)(508600001)(6486002)(966005)(82960400001)(66476007)(26005)(186003)(66946007)(316002)(5660300002)(8676002)(86362001)(4326008);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?c3B5MjkyRmdaNURuSS9HbWxtT2pDTHNXeG03Mi9UakFiSk9vNkJQVEE1RzNr?= =?utf-8?B?SmRsemt6ME5RaGlJRjZPcC9qcWVDajJGUXEyRmdwTWNzZTh0NFJnd1dva3N3?= =?utf-8?B?K0RmMS8wcjFSSGJtQkxYcmcrSERtOVVFNmRkWTFuY3VERWtPRlZ1ZzZTaDcz?= =?utf-8?B?ZVhxSjFUbGxTMUJWK2VxdnNpV1lpTmJTVXBjWW52WXJpYkd5NFZsRjdLQThW?= =?utf-8?B?TWVCWXV4VTV4d2NNLzFpb3hvZ0d3dTRmbDdLeFBRZzJHanVReW1LZzBQWTJ5?= =?utf-8?B?QVh4Q01kelFXRWZxRU1sTjlra3ZRdzJwWCtCYmhwSy9STUNNdmxIYis3dklV?= =?utf-8?B?dG04M1BHQXV6OXlUZTNLT3ZTdEYycUJvYmV4dDExbWNOUTI3djlmZjgyd3pK?= =?utf-8?B?eDFwUUd5K0w2UytyUDF1cXNNNUJ0UnlBalIyWUErWUx5UzdCbEVUZHA5YjZM?= =?utf-8?B?bk5tb1NpL3l1MTkzMEh6M2FwNFNoMVdHR1dsd0VDamxUY3ZTWkNyTDVTekk0?= =?utf-8?B?eUF1QTU5QnFTenJrYkx5N1VtdVNuRWFYdzVOdEFJMVFZUHIvTmhXMHJ1MEpK?= =?utf-8?B?a2NsRlV1R0Q3VlYzRnFHZTFzQWtsU3F1MHZjTHpJNFcyaGtmZmhqZHBxVk5p?= =?utf-8?B?WVhlOFVkRGRxbHZjVzJ6cjZEaWtWcXhza3VkOERubko1eFFER2lqdG1tSUVM?= =?utf-8?B?QTR6UThrVGZmSzJPNDJRZG9Pc1p6bXFoSTVDS0FGZUZEdkovRmJNbk9jT1dC?= =?utf-8?B?c0JlcllHSTZkTkhSSmRSSnFqYk4vcE5STWk4L1V4UWFhZUlLUVRSTjZsa1R3?= =?utf-8?B?MTNtMDFvU0tCQkV4OUxmT2pIVXBkYXZUanJ2TWVaQStuZGlLVmZlTkhRSEJP?= =?utf-8?B?T29HbmpqeG51UmhFbUZSNGVkNHRJaTYyRWNDK0FMb0RmdWVuVDN6N3NhTy9P?= =?utf-8?B?T0UxR3VUWE1TM3ArMm1GTG90TEp0d3NWa3p3eUpvejhXV1lnN1FxNjFqZDJX?= =?utf-8?B?UTR6WFgwL2ZyQnUxTytubjlDNDBMSkREVWREeXRGbzg2cTNyeHRFVm9icTF5?= =?utf-8?B?TGQ3UjNEekNHamwweUdxUjhnd3VkNWdYaU5kVXNoejZiM3dNYi9ObU9rTVdi?= =?utf-8?B?UVhXRmNYSXR2L3pJYzlueFRwem0wTTl3Vkpmdzg2ald6UllUMlB0VkxXMjA5?= =?utf-8?B?Nk5vQ1lZaSt4aVc4TEZpZVZ1Sks4a1FVUS90c294eW5NaEtJdEEydk5RZE1I?= =?utf-8?B?S3JMZ0s1aE9vVHNHaW12TXdOTlMxNFYxNk1ZYmJRZHE4dVg0ZmZFQWFuaVEw?= =?utf-8?B?QlI0MUk3NUhycmlBZ3FmS1JhN1FjN3FzbkZBM0hFcG83WmZlbjhjNThqYnRn?= =?utf-8?B?QldOTHJyaUF1dTIwZnNFK0lTQk5uMU5PYVEydkxBem5xaS9ORVc2aWgxWmlN?= =?utf-8?B?UWZsTDRPVnM3Nm5IK3R1d0gvTURjcWpveFBub2xxQkpwZHNORHJRL0tiVzBS?= =?utf-8?B?S1BWNmFIRVFMbmQwWC9veUVVdjdpTXdwdG1SVmEyd0FrUGlwRzhlSlQ2eXpM?= =?utf-8?B?QkJqNUVkZVNTT2NZZVEvcGVWYjdVQUdVVlFhVWF3TzFzc0dibXoxRnFoUjRS?= =?utf-8?B?ci9LMjBEbHRMd3R5ZEtKMkZ1WXVWVGZha2NkQmorK1pncnNVQnpLZ2V6NkhN?= =?utf-8?B?Vndoa3IwSVFhKzFCY2QxbHJ6NnI0eEhicHB4MldsUnBkUDdNUU5BOHlqdjVu?= =?utf-8?B?ZEt6U0U0T05RWXl5RkthNXlSSHJiMUZOa0d1OE1UQjJVaG1ZZXhuaEZiZERP?= =?utf-8?B?WTlKWWp6NmJNT3cxR3ZnTWFjNXVuSVdKbUJxdWUvYWZmZ0hKOFhUTzZUdWVL?= =?utf-8?B?MmdqQ1B4WEpyV2IyQnhyVFpsLzBUR3dBaUxyYUVROEtTKzVsTm5VMVN5MnpW?= =?utf-8?B?RXF1bDRlaWdvdWtQUS9vYkdINHVuYVh4V3FEVFAwQmFobnRtbFdsdEx2N0VX?= =?utf-8?B?REpGOVowU2xTS213M1R3QjZVMXRTdVQ1N05Fbzh2V0tlRjhRaWFsL3Jqdzda?= =?utf-8?B?VmdiUXArQmFBK1dnNEk2L1RXQ2xVNW1PNlVHOXhIT2FCMTlYaDh6TEFIZVRl?= =?utf-8?B?bE5ZbmhGbmx0UHRnQmpRVEtmdU5QVnN1TExndk5qTCs2Z1VBZmR3bHZhSkU2?= =?utf-8?B?czhWNm1ZYWg2bEJPOUxXakp1Skc1d2VSYi8wT3pBQkVKT3ozVkZ1UTRZaEJY?= =?utf-8?B?Yml5WmlPcTltWHNaaEphVHhDSUVlQ0ttV3JnVlRvZ01wUFZmYmxLdjhUMlhG?= =?utf-8?B?V3FDMzhwaGRrbkt5OXpyOWgwRGVJa0JoamNTYTNacUhKUjh6NEZkUFlrRUc1?= =?utf-8?Q?CpIXQxddWsNuHdLc=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: 5552b399-a954-4010-7b4b-08da1deb4fd2 X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Apr 2022 07:49:32.0842 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Ny5iNYPzmm1zn8PIaVMm12E1Flm84UB7yGQhlK6XvlR7jPdWXbDae4BK8DMmC2Z4UOyDN5NDKLin4y4BS1kawA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR03MB3948 X-OriginatorOrg: citrix.com X-Rspamd-Queue-Id: 4KfBSP2Bw1z4tc3 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=citrix.com header.s=securemail header.b="fcbV/ENY"; dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=fjOZxa2p; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=reject) header.from=citrix.com; spf=pass (mx1.freebsd.org: domain of "prvs=09619e58f=roger.pau@citrix.com" designates 216.71.145.142 as permitted sender) smtp.mailfrom="prvs=09619e58f=roger.pau@citrix.com" X-Spamd-Result: default: False [-5.64 / 15.00]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+exists:216.71.145.142.spf.hc3370-68.iphmx.com]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+]; DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject]; RCVD_IN_DNSWL_MED(-0.20)[216.71.145.142:from]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_COUNT_ZERO(0.00)[0]; MIME_TRACE(0.00)[0:+]; R_MIXED_CHARSET(0.56)[subject]; ASN(0.00)[asn:16417, ipnet:216.71.145.0/24, country:US]; FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=09619e58f=roger.pau@citrix.com]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; FORGED_SENDER_VERP_SRS(0.00)[]; DWL_DNSWL_LOW(-1.00)[citrix.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[freebsd-xen]; MID_RHS_NOT_FQDN(0.50)[] X-ThisMailContainsUnwantedMimeParts: N On Thu, Apr 14, 2022 at 10:20:25AM +0300, Ze Dupsys wrote: > On 2022.04.05. 18:22, Roger Pau Monné wrote: > > I've pushed the changes to: > > > > http://xenbits.xen.org/gitweb/?p=people/royger/freebsd.git;a=shortlog;h=refs/heads/for-leak > > > > (This is on top of main branch). > > > > I'm also attaching the two patches on this email. > > > > Let me know if those make a difference to stabilize the system. > > > > I do not know should i start a new thread, but i have captured another > panic, new trace, this is on different machine, similar setup, RELEASE-13.0 > + 2 mentioned patches. > > I do not know how to reliably repeat it, nor the cause. But i have suspicion > that this happens when doing some of steps like: create new ZVOL, turn one > VM off, add new HDD/ZVOL path to VM in cfg file, start VM back up, inside > this VM do some HDD load on newly added HDD (install stuff, extract data, > etc.) + something of: shut all VMs down one by one, then do init 0 or 6, or > create new other VM. On this machine i can't experiment too much, no serial > output available either. So you haven't seen this panic with the 3rd patch applied? I guess that's also possible because when testing the 3rd patch you are using a HEAD kernel rather than stable/13, so the ZFS code might have changed. > > Fatal trap 12: page fault while in kernel mode > cpuid = 3; apic id = 06 > fault virtual address = 0x68 > fault code = supervisor read data, page not present > instruction pointer = 0x20:0xffffffff821dc99d > stack pointer = 0x28:0xfffffe00c6b497d0 > frame pointer = 0x28:0xfffffe00c6b49870 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 0 (xbbd26 taskq) > trap number = 12 > panic: page fault > cpuid = 3 > time = 1649915274 > KDB: stack backtrace: > #0 0xffffffff80c57385 at kdb_backtrace+0x65 > #1 0xffffffff80c09d61 at vpanic+0x181 > #2 0xffffffff80c09bd3 at panic+0x43 > #3 0xffffffff8108b187 at trap+0xbc7 > #4 0xffffffff8108b1df at trap+0xc1f > #5 0xffffffff8108a83d at trap+0x27d > #6 0xffffffff81061818 at calltrap+0x8 > #7 0xffffffff821c035a at dmu_read+0x2a > #8 0xffffffff8218da3a at zvol_geom_bio_strategy+0x2aa > #9 0xffffffff80a7f074 at xbd_instance_create+0xa3d4 > #10 0xffffffff80a7b00a at xbd_instance_create+0x636a > #11 0xffffffff80c6b021 at taskqueue_run+0x2a1 > #12 0xffffffff80c6c33c at taskqueue_thread_loop+0xac > #13 0xffffffff80bc7c9e at fork_exit+0x7e > #14 0xffffffff8106289e at fork_trampoline+0xe > Uptime: 24m0s > (ada0:ahcich0:0:0:0): spin-down > (ada1:ahcich1:0:0:0): spin-down > (ada2:ahcich2:0:0:0): spin-down > Dumping 2922 out of 6104 > > > > cat panic.log| sed -Ee 's/^#[0-9]* //' -e 's/ .*//' | xargs addr2line -e > /usr/lib/debug/boot/kernel/kernel.debug > /usr/src/sys/kern/subr_bus.c:2410 > /usr/src/sys/kern/kern_racct.c:632 > /usr/src/sys/kern/kern_racct.c:617 > /usr/src/sys/dev/isci/isci_sysctl.c:92 > /usr/src/sys/dev/isci/isci_sysctl.c:0 > /usr/src/sys/dev/isci/isci_oem_parameters.c:130 > /usr/src/sys/dev/hyperv/input/hv_kbd.c:540 > ??:0 > ??:0 > /usr/src/sys/dev/xen/blkback/blkback.c:3083 > /usr/src/sys/xen/xenbus/xenbusvar.h:96 > /usr/src/sys/kern/subr_kobj.c:145 > /usr/src/sys/kern/subr_module.c:255 > /usr/src/sys/kern/kern_event.c:0 > /usr/src/sys/dev/hyperv/pcib/vmbus_pcib.c:1158 > > > Full output of (kgdb) backtrace > #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 > #1 doadump (textdump=) at > /usr/src/sys/kern/kern_shutdown.c:399 > #2 0xffffffff80c09956 in kern_reboot (howto=260) at > /usr/src/sys/kern/kern_shutdown.c:486 > #3 0xffffffff80c09dd0 in vpanic (fmt=, ap=) > at /usr/src/sys/kern/kern_shutdown.c:919 > #4 0xffffffff80c09bd3 in panic (fmt=) at > /usr/src/sys/kern/kern_shutdown.c:843 > #5 0xffffffff8108b187 in trap_fatal (frame=0xfffffe00c6b49710, eva=104) at > /usr/src/sys/amd64/amd64/trap.c:915 > #6 0xffffffff8108b1df in trap_pfault (frame=frame@entry=0xfffffe00c6b49710, > usermode=false, signo=, signo@entry=0x0, ucode= out>, ucode@entry=0x0) at /usr/src/sys/amd64/amd64/trap.c:732 > #7 0xffffffff8108a83d in trap (frame=0xfffffe00c6b49710) at > /usr/src/sys/amd64/amd64/trap.c:398 > #8 > #9 0xffffffff821dc99d in dbuf_write_children_ready (zio=, > buf=, vdb=0x0) at > /usr/src/sys/contrib/openzfs/module/zfs/dbuf.c:4642 If this trace is correct the error is from passing vdb == NULL to dbuf_write_children_ready(): https://cgit.freebsd.org/src/tree/sys/contrib/openzfs/module/zfs/dbuf.c?h=stable/13#n4551 The function will unconditionally dereference (v)db, so passing NULL will trigger a page fault. I have no idea however how can you get to this state. Might be worth posting the trace to freebsd-fs@freebsd.org in order to get some feedback from the ZFS people. It's possible the issue is with blkback, but I would benefit from some help about what's wrong with the data I'm providing to d_strategy. Please Cc me on the email if you send to freebsd-fs@. Thanks, Roger.