From nobody Wed Mar 02 13:57:12 2022 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4B37619ED953 for ; Wed, 2 Mar 2022 13:57:30 +0000 (UTC) (envelope-from prvs=0530a50c4=roger.pau@citrix.com) Received: from esa5.hc3370-68.iphmx.com (esa5.hc3370-68.iphmx.com [216.71.155.168]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4K7wfd0wLlz52MV for ; Wed, 2 Mar 2022 13:57:29 +0000 (UTC) (envelope-from prvs=0530a50c4=roger.pau@citrix.com) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1646229449; h=date:from:to:cc:subject:message-id:references: in-reply-to:mime-version; bh=V6cL1ewHKKCqY5LNPUEBwGw6g6/GzIhBP4PA1PXw0tU=; b=eKwlf1BEl2jh2eDiQtEPZmIj1/yfro81MdG3ux7mFV7T7y31NVG68P3i D5DqgMzpZaJ2h22K0fo9TrCpDst24sQidIxAdNbJwwiPmmTfH7wwpoVYj 5gtI/g37083Acz3UnQ+Vq9xnS7/JaSF4UQuLBH//0JRVXRS//3Wx/SM6R 8=; X-SBRS: 5.1 X-MesageID: 64756999 X-Ironport-Server: esa5.hc3370-68.iphmx.com X-Remote-IP: 162.221.156.83 X-Policy: $RELAYED IronPort-Data: A9a23:ekYKzqmcD8Zz4fljhDEzJOjo5gyQJkRdPkR7XQ2eYbSJt1+Wr1Gzt xIaC27TOq2PajT3c94kaou+oEhT656Eyt43TABr+Cg3QyMWpZLJC+rCIxarNUt+DCFioGGLT Sk6QoOdRCzhZiaE/n9BCpC48T8kk/vgqoPUUIYoAAgoLeNfYHpn2EoLd9IR2NYy24DiW1nV4 rsenuWEULOb828sWo4rw/rrRCNH5JwebxtB4zTSzdgS1LPvvyF94KA3fMldHFOhKmVgJcaoR v6r8V2M1jixEyHBqD+Suu2TnkUiGtY+NOUV45Zcc/DKbhNq/kTe3kunXRa1hIg+ZzihxrhMJ NtxWZOYUiIqYvXOmv0keFoJSwNRbJB+2Zj1Li3q2SCT5xWun3rExvxvCAc9PJEC+/YxCmZLn RAaAGlTNFbZ3bvwme/lDLk37iggBJCD0Ic3oHZvwCufFf87aZvCX7/L9ZlT2zJYasVmQ6qBO pRIOGsHgBLoezx2A3gwC5cEo+L3vlv4KD4Ag1HInP9ii4TU5FMoi+W8WDbPQfSAWMVPmVmwo GzM9G39RBYAO7S3xyCI/3i3iqrfkCXTVpgfGbej+rhtmlL77n4eBAAOTkqgifa8g0+6HdlYL iQpFjEG9PZoshbxF5+kAkP+8CXsUgMgt8R4CMg5tFmPyLPtyEXIXkorHwN5SfsUu5pjLdA17 WOhk9TsDD1plbSaT3OB67uZxQ+P1TgpwXwqPnFdE1ZcizX3iMRq10+UEI4/eEKgpoCtQVnNL ya2QD/Sbln5peoCzO2F8F/OmFpATbCZH1dutm07so9Ihz6VhbJJhaT1sTA3Dt4ade51q2VtW lBewaByC8hUUPmweNSlGrllIV1Qz6/t3MfgqVBuBYI90D+m5mSue4tdiBknehs3a5hcIW63O R6J0e+02HO1FCHwBUOQS9jsY/nGMIC6TYi1PhwqRoAmjmdNmP+vo3g1OB/4M5HFm0kwi6AvU ap3gu73ZUv2/Z9PlWLsL89EiOdD7nlnmQv7GMCqpzz6gOH2TCPEFt843K6mM7lRAFWs+16Or b6y9qKiln1ibQEJSnKPoN5LcAtTdiBT6FKfg5U/S9Nv6zFOQQkJI/TQ3akga8pimaFUnf3P5 XazRglTz1+XuJENAVjihqxLAF83YatCkA== IronPort-HdrOrdr: A9a23:3eg78a4osg98Xcom4wPXwSSBI+orL9Y04lQ7vn2ZFiY7TiXIra yTdaoguCMc6AxxZJkh8erwX5VoZUmsj6KdgLNhRotKOTOJhILGFvAB0WKP+UyEJ8S6zJ8h6U 4CSdkBNDSTNykCsS+S2mDReLxBsbq6GeKT9J/jJh9WPH5XgspbnmFE42igYylLrF4sP+tEKH PQ3LsPmxOQPVAsKuirDHgMWObO4/XNiZLdeBYDQzoq8hOHgz+E4KPzV0Hw5GZUbxp/hZMZtU TVmQ3w4auu99m91x/nzmfWq7BbgsHoxNdvDNGFzuIVNjLvoAC1Y5kJYczLgBkF5MWUrHo6mt jFpBkte+x19nPqZ2mw5SDg3gHxuQxen0PK+Bu9uz/OsMb5TDU1B45qnoRCaCbU7EImoZVVzL 9L93jxjesaMTrw2ADGo/TYXRBjkUS55VA4l/QIsnBZWYwCLJdMsI0k+l9PGptoJlO21GkeKp ghMCjg3ocWTbvDBEqp/lWHgebcFEjbJy32DXTr4aeuontrdHMQ9Tps+CVQpAZDyHsHceg12w 31CNUYqFhwdL5kUUtcPpZ3fSLlMB26ffrzWFjiU2gPUpt3fk7wlw== X-IronPort-AV: E=Sophos;i="5.90,149,1643691600"; d="scan'208";a="64756999" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nvd+14Iy1FRiJ5bhfCqZ6g6iPC24MS8jQMuUNRTO59M40Adidyrnnhi3/qtkM2w4WCxucmwnuIgSBXVdEbMqfJZk2aOL978stpP/tBk/uZ/y87k0kKTceBAYvjDiZXYuw1leG6CNWmOsrBVUGxIY04tMmIoTuTap3Z5DSuTT/x/yvo2dKLgKecU82oVlqNNJfJKVR4l5oEN1oswknZKpPwm2RoISQG88F0qYR7Oep/QvKtBPWMAgHtYG+m4kRiZDs0r6Id1rIoBWqZasPW/YIW9WQOT/MWkVB8EPrNV8T6nTPephT5MaaRSrwJowXxO1XOMPKTWAhF8vipd1rjSlpg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=sF4YLlDfeMxNEfZ4py2VdNjsr0DF+89ztOd7wZWDvlg=; b=eU71q0rZBSEivNjpS02x0pisq1fUZQjBwhy61BRIXEx5EtHx40AK5POT9X5Mwptf2nxFkJARGzSq9C3jWfn52jmzVCKmWCm2CNHrgHa5o1+PK8wzw5d1iFJ8tZnsDtYZnhWLsrKybITj7eJZpESWrY7BiZDQ3uQfQia46LdeKNKBu4YBmaw9zw6HXifBiINU3dR5Xrb3totLnJBH6uFVq3j63DOMvixEY3D2WQUfaXuGhOAbERQPYZFD6AzG7MICpH9VmxhwJPKEVGC144TUaH6ortnTCM3PBF+vzGBWs9RR5YXDKvRIXziHL/P5Tw8NZ8gDD0Y0bKNx+qJlVZrAig== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com; dkim=pass header.d=citrix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=sF4YLlDfeMxNEfZ4py2VdNjsr0DF+89ztOd7wZWDvlg=; b=u/D3RKEik2w0eijdnu6rCfVhyG9St4KFqmtVI8Umle/JRIB+GJvYc/GSlGB0UXrqumKR+cg88AdJ/Fpk43bS4WW9MkfcdV23LnY1qRq4R4gBRiV/8uC6a++B1aFfspF3XhjckOhAnICljQQ6I0AcuNmG8iAKSsxeqDNYPWE3MdM= Date: Wed, 2 Mar 2022 14:57:12 +0100 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Ze Dupsys CC: , Subject: Re: ZFS + FreeBSD XEN dom0 panic Message-ID: References: <202203011540.221FeR4f028103@nfbcal.org> Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: LO4P123CA0242.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1a7::13) To DS7PR03MB5608.namprd03.prod.outlook.com (2603:10b6:5:2c9::18) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 6805a47b-fc87-40b4-8267-08d9fc548ffe X-MS-TrafficTypeDiagnostic: DM6PR03MB4026:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Mf5UKNEW6jsrSuiM1g1NSZYyDsWsBJhauH00G0slMxPjIRCp+Dg9wYiseiVnnnR2UzdiwXA1XvgzEYz8hit5Nlz9yAaa3V509AknXjEpZRtx7AqCzOFC4C9xB/weyF5FGaSlOSXjr0ZZHKDDm/E0PHLHRsKw5dwSLAkFx8KIMlujOVXo6iadZ8mqSGY7kzF65+bG0koxqZah0YqfOJ06mu/tzSE8yp6en6pdKHcJwBEiHO8uZXEnCvWfr+IzfUG2fLldi6ksMhb4X2qLE70cEK0008/bBVsecRwrKrYwwP+ouIhd/WFdaMCyjLp5QdF32TID99NnA8lT3oaeCubj6jt6gl1eMDgn11DzqTpWNDUfqdf0zHyBszxIufaF+tQqIe0oJ8OZLNs10B6K6zDFtWfYjRP1y6e7lkj8tdV8FZS5kAyskQ7hOWlG0NrdLVOk9hWS47jbyo6fFhqIMShzt3ewVkWoDAVDgqGTYPCc1xXaYqYWxFtRHmJLWMLGs2DDSmYWVP0LXaiPwrx4qi88NLIgMIVydMPwBOkapHXVwe/EA+XzZaPT/5YXAVThmX98MY8sif/lhTI31+hIcDanWDLgh+Qly3gopEm3n/LKmc2nWpP/zsllhPu3IVZnciriMP0zA1k+XgEpMNk12Hp9cA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(7916004)(4636009)(366004)(316002)(6512007)(9686003)(5660300002)(38100700002)(8936002)(86362001)(8676002)(66556008)(66476007)(66946007)(4326008)(2906002)(186003)(26005)(82960400001)(85182001)(6506007)(6486002)(6666004)(508600001)(6916009)(83380400001)(33716001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?YjJxeFNuTll0VFpSRFd3SXVtRzJCdkdBTE04S0QvOGJmYlJid3Jockk4U3JT?= =?utf-8?B?MjVlUGlVbkFsdVZMOTVJRmFGUVFGWXhXK0phZ09JSDBlZVIvYUwwbkNYSS9s?= =?utf-8?B?eVpVUXkzeTNjdVBIY2Q0NmxQWDJnSW5QQlZMTy9EL1A2ZStHbUE2Y1N0N1ZH?= =?utf-8?B?WlRUWlkxWnlnOGkrcjQ5NThFMU9tdnhWaG80eHR3dCtqbHZzekJBWUR1Wjd0?= =?utf-8?B?a2pTcUhzZjA4QityQ3p5SnJHTDRRdjM4bUVDM29ndS9NUW5sRXZZTElPQysr?= =?utf-8?B?QVpkazJpcTlRQjA4djI2dW1VMEljTkVTNVRZZGV0c3hwenFQTWJtVVRBMWp3?= =?utf-8?B?RkpKSWRYUll1cWRFdFNaZ0NOUDIzcElHUmVhamR0MStKMzJ6K3lmdVJIaGl5?= =?utf-8?B?a1JSVXk3TitSODc2TWNkS2xpOWxTa3diVnl0YVRwMGNWbDFkVm9vaFVOVURE?= =?utf-8?B?VjYxZTgvMDFnN1FHMkpyS2pVNUQySUtCTnVCWktJdkRNRlczOXoyVWtGT21X?= =?utf-8?B?aUVKS2RaMTJjYTJjZC96ZkQrbDltaXY0bFVVRE13dnM1SkpURXFmQUFZRXhW?= =?utf-8?B?U2N4ZmRyWHgzdkRNVGtpOVQyeTN3VUNKZlZBalBTQUNlaVcrYmJSd0E0N29m?= =?utf-8?B?MXByREltcmhueVptNUpSRWc1dFlka252NWh5MmtUaDV2b2JTeHlnajNKdVlG?= =?utf-8?B?WE1vbmpJOGpySE9iRHY3encxSUNoZXJ5QzJrTGlBQ25rZzRiNlJ3QTNKRlcz?= =?utf-8?B?MlhSaUxRTThKdlVVWDNQWTNmQUhPbHY2RnlEUTdab2tOUkxmK1IwTzBqMXho?= =?utf-8?B?ODNDWEovN3B0OXI4T1IwSHUrQVZKclFubWxvNExuUU9FSmFMYzJqQ3JLYnd6?= =?utf-8?B?cW80VCtUQUJyc3Y4UUJZZCtPSUR2TGJpT2dCUUdndlNQYzFiYXlXWFNrTDlK?= =?utf-8?B?ckI4ZE4vekYrdGhqTnV5UWhtSEludDM5S0JhQzMwNmRNZEhxVHBjRGd4UDk5?= =?utf-8?B?OTlCNWxRZWVkd3hIeC9FdkxZY0hqdEIwcjludFdvdElBam5YYUkyUlMwaWg5?= =?utf-8?B?NXRsWjNaVjlGVUsvNXlIcWUzakREU01VcURzN1E4Y2huZDRBdFd1SGtLc0hp?= =?utf-8?B?YU1PNDM0UGdiSUJjMVY5cThNa01SUlE3aFpLcnFzSzhxWWJRUmsyTU50STgz?= =?utf-8?B?RE9RV0lCbnlldGdEK1ROVXBZbElQeWwxZzJBWWVQekJCRmxwT2lNelNPcGRu?= =?utf-8?B?SVhxMjFrNSt4dGRkMmN0MWE3UmYwdHFqNmtmOFdneGM5U0dJVmM1Mm1TRnAy?= =?utf-8?B?K0NOOEx3Wlp3cHJNc3RsV0UxSHVBTjdXQ2N5VXFCSENhNmVES3F0NGlJRWJj?= =?utf-8?B?NDRHOW1NZXpYSzNiWGpsQmJJaEYwZmFma240WU1jNklkbVB0eFhQKzZyY1JT?= =?utf-8?B?WHBqbURKOVV3cnQ1dWFsRGE1ZlZzeWdXRnZJMWdQVXlUc0VBd1Q5ZWdWYjUv?= =?utf-8?B?dzZXS2MxY2pDYkVsUTdxa3AwOE1KZEhMbTc1RzRibU5zVGlqWDJOTU8yc1VX?= =?utf-8?B?WHZtRGVEU09jaW9iRGMrRGZORmE4a1pxcURxYVJxR3dyN09NWmcvUHZVN0M0?= =?utf-8?B?OGZyOWtDbVNMM001UDJueDg5SStkWjc1TGNScXV4SlJGYnllRjBDWjJHRUVI?= =?utf-8?B?L3lnSjhDdVpqb0toOXpocCt3dU5GQURQVGdaRG0ybTBwcTJzWldoZ3MreU1B?= =?utf-8?B?aXcxd1VOQVRYOXNnZmRzWkNLTzUyNDR5d1EzdWs0OHgrRitLbDhiaFBlQ0VX?= =?utf-8?B?NmZGOVhoajlFU05ZQUJ2QVdJOWxNTGJudTJVVHVxZUxyc0x6bnI3c2dnVW9t?= =?utf-8?B?UUZzblh1OG1XUThScW5mbitnNy8wcVZGNXpGMDkrOTJheUNIT2pFekNGbDZY?= =?utf-8?B?VnpqNVlBd0c0aW9EcndhSTRBanViNVZvaUxndzY0MWVIRXRqbkFTQVRzRlE1?= =?utf-8?B?VmhOM21BWHdIaitGZ2lRUE95cWl3bHdUMjhhRzdITjIwNG1rVkQxeHFuN1pM?= =?utf-8?B?UjVadkVMWnVxbG5CbHg0Q2ZKZDhqZ2F5YkJPR1dMcXBoSVFCc3hpNkptTWtQ?= =?utf-8?B?ZU9sZkIxc296SWVHM3VyTittblJUdXcrRWl6USsyK09PaC8vTEk0aE4rMGRa?= =?utf-8?Q?LsBml+zmbWUN3g+gl+0lWyM=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: 6805a47b-fc87-40b4-8267-08d9fc548ffe X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Mar 2022 13:57:17.4628 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: wLrLFm2n6eFga8ywXSoJU25XmsSIjaoJwq/24++3nxauyEnQFc0x/G/jJmTqc/4l3+qXEZ1dujJpnXfPTS45bw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR03MB4026 X-OriginatorOrg: citrix.com X-Rspamd-Queue-Id: 4K7wfd0wLlz52MV X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=citrix.com header.s=securemail header.b=eKwlf1BE; dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b="u/D3RKEi"; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=reject) header.from=citrix.com; spf=pass (mx1.freebsd.org: domain of "prvs=0530a50c4=roger.pau@citrix.com" designates 216.71.155.168 as permitted sender) smtp.mailfrom="prvs=0530a50c4=roger.pau@citrix.com" X-Spamd-Result: default: False [-5.64 / 15.00]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+exists:216.71.155.168.spf.hc3370-68.iphmx.com]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; RCVD_IN_DNSWL_MED(-0.20)[216.71.155.168:from]; DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+]; DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_COUNT_ZERO(0.00)[0]; MIME_TRACE(0.00)[0:+]; R_MIXED_CHARSET(0.56)[subject]; ASN(0.00)[asn:16417, ipnet:216.71.154.0/23, country:US]; FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=0530a50c4=roger.pau@citrix.com]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; FORGED_SENDER_VERP_SRS(0.00)[]; DWL_DNSWL_LOW(-1.00)[citrix.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[freebsd-xen]; RWL_MAILSPIKE_POSSIBLE(0.00)[216.71.155.168:from]; MID_RHS_NOT_FQDN(0.50)[] X-ThisMailContainsUnwantedMimeParts: N On Wed, Mar 02, 2022 at 10:57:37AM +0200, Ze Dupsys wrote: > Hello, > > I started using XEN on one pre-production machine (with aim to use later in > production) with 12.2, but since it experienced random crashes i updated to > 13.0 in hope that errors might disappear. > > I do not know how detailed should i write, so that this email is not too > long, but gives enough info. > > FreeBSD Dom0 is installed on ZFS, somewhat basic install, IPFW and rules > for NATting are used. Zpool is composed of 2 mirrored disks. There is a > ZVOL volmode=dev for each VM and VM's jail that are attached as raw devices > to DomU. At the moment DomUs contain FreeBSD, some 12.0 to 13.0, UFS, with > VNET jails, epairs all bridged to DomU's xn0 interface. On Dom0 i have > bridge interfaces, where DomU's are connected depending on their > "zone/network", those that have allowed outgoing connections are NATted by > IPFW on specific physical NIC and IP. So from the traces on the ticket: panic: pmap_growkernel: no memory to grow kernel cpuid = 0 time = 1646123072 KDB: stack backtrace: #0 0xffffffff80c57525 at kdb_backtrace+0x65 #1 0xffffffff80c09f01 at vpanic+0x181 #2 0xffffffff80c09d73 at panic+0x43 #3 0xffffffff81073eed at pmap_growkernel+0x27d #4 0xffffffff80f2dae8 at vm_map_insert+0x248 #5 0xffffffff80f30249 at vm_map_find+0x549 #6 0xffffffff80f2bf76 at kmem_init+0x226 #7 0xffffffff80c73341 at vmem_xalloc+0xcb1 #8 0xffffffff80c72c3b at vmem_xalloc+0x5ab #9 0xffffffff80f2bfce at kmem_init+0x27e #10 0xffffffff80c73341 at vmem_xalloc+0xcb1 #11 0xffffffff80c72c3b at vmem_xalloc+0x5ab #12 0xffffffff80c72646 at vmem_alloc+0x46 #13 0xffffffff80f2b616 at kmem_malloc_domainset+0x96 #14 0xffffffff80f21a2a at uma_prealloc+0x23a #15 0xffffffff80f235de at sysctl_handle_uma_zone_cur+0xe2e #16 0xffffffff80f1f6af at uma_set_align+0x8f #17 0xffffffff82463362 at abd_borrow_buf_copy+0x22 Uptime: 4m9s Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x22710028 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff80c45892 stack pointer = 0x28:0xfffffe0096600930 frame pointer = 0x28:0xfffffe0096600930 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 1496 (devmatch) trap number = 12 panic: page fault cpuid = 0 time = 1646123791 KDB: stack backtrace: #0 0xffffffff80c57525 at kdb_backtrace+0x65 #1 0xffffffff80c09f01 at vpanic+0x181 #2 0xffffffff80c09d73 at panic+0x43 #3 0xffffffff8108b1a7 at trap+0xbc7 #4 0xffffffff8108b1ff at trap+0xc1f #5 0xffffffff8108a85d at trap+0x27d #6 0xffffffff81061b18 at calltrap+0x8 #7 0xffffffff80c62011 at rman_is_region_manager+0x241 #8 0xffffffff80c1a051 at sbuf_new_for_sysctl+0x101 #9 0xffffffff80c1949c at kernel_sysctl+0x43c #10 0xffffffff80c19b13 at userland_sysctl+0x173 #11 0xffffffff80c1995f at sys___sysctl+0x5f #12 0xffffffff8108baac at amd64_syscall+0x10c #13 0xffffffff8106243e at Xfast_syscall+0xfe Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 02 fault virtual address = 0x68 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff824a599d stack pointer = 0x28:0xfffffe00b1e27910 frame pointer = 0x28:0xfffffe00b1e279b0 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 0 (xbbd7 taskq) trap number = 12 panic: page fault cpuid = 1 time = 1646122723 KDB: stack backtrace: #0 0xffffffff80c57525 at kdb_backtrace+0x65 #1 0xffffffff80c09f01 at vpanic+0x181 #2 0xffffffff80c09d73 at panic+0x43 #3 0xffffffff8108b1a7 at trap+0xbc7 #4 0xffffffff8108b1ff at trap+0xc1f #5 0xffffffff8108a85d at trap+0x27d #6 0xffffffff81061b18 at calltrap+0x8 #7 0xffffffff8248935a at dmu_read+0x2a #8 0xffffffff82456a3a at zvol_geom_bio_strategy+0x2aa #9 0xffffffff80a7f214 at xbd_instance_create+0xa394 #10 0xffffffff80a7b1ea at xbd_instance_create+0x636a #11 0xffffffff80c6b1c1 at taskqueue_run+0x2a1 #12 0xffffffff80c6c4dc at taskqueue_thread_loop+0xac #13 0xffffffff80bc7e3e at fork_exit+0x7e #14 0xffffffff81062b9e at fork_trampoline+0xe Uptime: 1h44m10s This all look to me like Out of Memory conditions, can you check with `top` whats going on with your memory? Might also be helpful to record periodic calls to `vmstat -m | sort -k 2 -r` to try to figure out what's using so much memory. Regards, Roger.