From nobody Thu Mar 24 16:26:22 2022 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 437CF1A235C0 for ; Thu, 24 Mar 2022 16:26:38 +0000 (UTC) (envelope-from prvs=07544547a=roger.pau@citrix.com) Received: from esa2.hc3370-68.iphmx.com (esa2.hc3370-68.iphmx.com [216.71.145.153]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KPVwY15Q7z4ll4 for ; Thu, 24 Mar 2022 16:26:37 +0000 (UTC) (envelope-from prvs=07544547a=roger.pau@citrix.com) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1648139197; h=date:from:to:cc:subject:message-id:references: content-transfer-encoding:in-reply-to:mime-version; bh=X11Ytl/Gh0k+aemNSzHDaCOFzveGcL2ikcPYe7Wkpu8=; b=fFtIoNOTXclglZptTC1vIUv4T83iS6GjvwfLlhE2wssAwi/iDUTR9tn/ ktCNzS7iuRV/xH8sdWVWpQDFuHi+xywXz4ZHpKAdWcA3Hg8rP9PfqhmdO vt70RNSAsgQLyyGEp1oizjOlXrEgVvVGinWZtFMMXpH9QRTWHm7JQ8B2z 8=; X-SBRS: 5.1 X-MesageID: 67093999 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.156.83 X-Policy: $RELAYED IronPort-Data: A9a23:5glWgKt+VqqHpCz9ccRMAinYWufnVEFeMUV32f8akzHdYApBsoF/q tZmKW+CbviNNzegKIx/bo+0/U0FupWHmNdmSAs9+Sg2Fy1G+JbJXdiXEBz9bniYRiHhoOOLz Cm8hv3odp1coqr0/0/1WlTZhSAgk/nOHNIQMcacUsxLbVYMpBwJ1FQyw4bVvqYy2YLjW1jX5 YuoyyHiEATNNwBcYzp8B52r8HuDjNyq0N/PlgVjDRzjlAa2e0g9VPrzF4noR5fLatA88tqBb /TC1NmEElbxpH/BPD8HfoHTKSXmSpaKVeSHZ+E/t6KK2nCurQRquko32WZ1he66RFxlkvgoo Oihu6BcRi82Y4rWk9kvSSJfKAB7EqRA4KXXGkag5Jn7I03uKxMAwt1rBUAye4YZ5vx2ESdF8 vlwxDIlN07ZwbjsmfTiF7cq1p9LwMrDZevzvll6yj7UF7A+SI3rSKTW/95Imjw3g6iiGN6AO JJIMms+PXwsZTVEFmUKKrNuhdylg3+geB1Uon2v/4cotj27IAtZj+G2bYu9lsaxbc9Inlyfu UrA8W7yCxVcM8aQoRKL6H+hjPTD2DP2XqocDrC1/OVmxlqJyQQ7GhAaSEOgsOKRhUu3WtYZI EsRkhfCtoBrqhbtFIOkGUTl/jjU5XbwRua8DcUltSaXwZfm+T2ePUkBYTdHcMY2j/EpEGlCO kCyo/vlAjlmsbuwQH2b96uJoT7aBRX5PVPudgdfE1JbvoCLTJUby0uWE409SPLdYsjdQ2mY/ tyckMQpa1z/Z+Yv3r7zw13IiinESnPhHl9svVW/so5IA2pEiG+Zi26AtACzARVodt/xory9U J4swZf2AAcmV83lqcB1aL9RdIxFHt7cWNEmvXZhHoM66xOm8GO5cIZb7VlWfRk1Yp5aIW+yP heO6Wu9AaO/2lPwPcebhKrrVqwXIVXIT4y5Bpg4kPIQCnSOSON31H43PhPBt4wcuEMtjbs+K f+mnTWEVh4n5VBc5GPuHY81iOZzrghnnD+7bc2rnnyPjOvFDFbIGOhtDbd7Rr1ghE9yiF6Oq Ig32grj40g3bdASlQGLrt9JcQFbdyRT6FKfg5U/S9Nv6zFOQQkJI/TQ3akga8pimaFUnf3P5 XazRglTz1+XuJENAVzihqxLAF83YatCkA== IronPort-HdrOrdr: A9a23:q19l1K9GhWGuDR3dgR1uk+C7I+orL9Y04lQ7vn2ZKCY0TiX8ra uTdZsguCMc5Ax6ZJhCo7G90de7Lk80nKQdibX5Vo3PYOCJggWVEL0= X-IronPort-AV: E=Sophos;i="5.90,207,1643691600"; d="scan'208";a="67093999" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=aTZj/IOfN5yxn4jqnuoZovleaUEd/XeOGQ+hmEAJRxhsJmLkDM+BsLxPmgZqK3XUIOMNbLSOv+mB3eTcglCb0HP63hu81YqT2xviDzWC+DTaExuNa5hlKy8WPRe+ehLaf+iJ3gwt/jZ6Gmb7/Wrx9y1OewmMn61J1osD9t8qNXmAbYwg/nYTJKlxiEghCQva/w/i8MRMJC1gimjkxcamMlc6Ho66LJCfTxWUL0suNf8soItp4FcutkH9KDk0V0sP8VejtW7sV/y6Sv/YGREQyhkPFjj0Wv2tuImJxJyk9u4MTG4sRvAVSdU4XmdcVgOx8sDNmFaL2syxLYw+izuwCg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=YQx+/hchdtV8Q+3fp9DsAbtZ44k6966PT/QWUWCY+vY=; b=D3oiH+3ASMifFaSsFJ9xg5p4K33zK5xk9TK8TxPdplUJjvPDZTrHpSxKsYD7zA4YibXivxIm5BqTaQB10wilVUE3UEgK7jfCNxZ+mduKgXCJ1bgUbZRHY9sFPRPqc5dp0iekoDxzBize4c4MKODWxSE/t3YErXtXRf0DsrF9IG04QCP9OlZMQ7QmBI32bg0fkSxj30k+jjvOrBXppgiTg7U/KEDWZwP3IZdzBaWlSkTx+BDO16knQGa7/eYDa0Gj0wv8qENlnkhf2YVF+roFcNA3ypEZTbHpqsq4kbkCWsym/idDBRCO68BnWPjHCS22aIIRPKzbQ/9YRgGF2n3+OQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com; dkim=pass header.d=citrix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=YQx+/hchdtV8Q+3fp9DsAbtZ44k6966PT/QWUWCY+vY=; b=rBgBXbiw2M9nf0ZhwX3m9Nl9B0GzQAMq7bVwKEESGHRTaSEhoFlU+qTHI71lh7O4liM9/AvEZG1c0gOnxjHIQlWXg27tBlEgt0icQjj+pKJn0wr+R1xJ00mq50s0tx/9N29UXakpXvMZGSJVNuqgvvKURmA4qAXoSymh2Zy9rwE= Date: Thu, 24 Mar 2022 17:26:22 +0100 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Ze Dupsys CC: , Subject: Re: ZFS + FreeBSD XEN dom0 panic Message-ID: References: Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-ClientProxiedBy: LO4P265CA0054.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:2ac::10) To DS7PR03MB5608.namprd03.prod.outlook.com (2603:10b6:5:2c9::18) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 36817cc5-e39b-40bc-8af4-08da0db30b96 X-MS-TrafficTypeDiagnostic: BYAPR03MB3527:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: JslAl9GqIGRPwpkkkvox/xMacGAek0pxwtZjZiXOs/8cyK3VQCaJmRRkP3LAOlYgPyEOchOP1EcpDlDjJx+4JQWxy5cFicF+nyQcgRAwIhRJuGKhTRFy+eabKuIQkvjSDhCyh6dS5PHvMWG0sxvRIRSvIaLGsmTx5rVil99+cgW4VzTWUJO2z+elBGvpqXiZBvcB20LwaxMiLmDx3Cjp/6JQZb1yODrDx2Jj3NqbWS3E3ymrpUsAUMTW39OcJWY00Qg7NEfQgqx/cFv7lEcJV69ctTOURqyAdwj0hWc6GaeYUrGFUKSQJk1/rMLTpnQpo8SD9sDXxOKeE+ck0uGGIXAqjUJQiJVVLKa7BxYbbc+s+aVCcnnOqwD1io3FGLTgFpFBBkDp11jgPlKqRaZIVoohcyR59Xs1H2/usYUYx9VX3sRXnEGo6ZUSaHITGgPplCtgMX12SOW9pE03NLlVIOsb5Ngw12A6GbeUzO0XPdmLy/aO88LG+ythylWuFYMpLPY21dPs5AI74HHlyATuoGlRNJoUb2lE6N9O1QB+NyT3EhyHi5DCrSezegoCdy4aVKxsbqDkv2AEV0ArHn1BEMOoGY3YpkkEIaNwA5bbpN/0b6ZPvPzicuk71KRYViiQKPb1nKCaPoppiML5/T7I/w== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(4636009)(7916004)(366004)(4326008)(2906002)(83380400001)(66476007)(8936002)(85182001)(66556008)(66946007)(8676002)(6512007)(6506007)(26005)(186003)(508600001)(6666004)(9686003)(33716001)(86362001)(82960400001)(5660300002)(6916009)(6486002)(38100700002)(316002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?OUR0MTNhZ1lZbEdlNUE3ODBZUVJzMExia1E5VXp1SXkzQ2lCVC93SkpyME1B?= =?utf-8?B?aThwekIrSzlHS2xMOHNFa1N4dVNpOTUzaWlPSmR2a3JGQXFDcXRXS2NkcEh0?= =?utf-8?B?Y1ZUc2tsTWEraE5ocmo5MFhhRDh4YVp0SVl0dDBvMVVWSXlHdEVNbXpoUzJi?= =?utf-8?B?czl2WUlURUtSTGxwU1huemgyZnBSeVFQVU9ObFpTdWNSVWhSUGtRR2N6VGVl?= =?utf-8?B?aTVVdElqVi9PRFZPUlFDN0FsNHo5L3J5bm5qMFN0NkZTYW92SDVpZlFrQ2Jp?= =?utf-8?B?b2JjTllBVzdwaDMxbWhBR2pLbUl2SzcvL0JCbXBKZzJadUllREI5dWNZWnRx?= =?utf-8?B?RFZzd2I2S1ZnWEttK3UrNVNZb3R0d3hQQmFzemg5LzZWRDFPL01LMi85UmNt?= =?utf-8?B?YzFFOWQ3YjhubkhGNDVPa3J5TG9nakg3dFgraHhhcnhZVjdqd01YOFFWMmtO?= =?utf-8?B?M25BbkdERmJGKzIzOU9Eb0lhUFdtc0M0U1VSVVRXcFRkbm9JNW1HM3BsVUUy?= =?utf-8?B?eG4vUk9nZC9FVXAzMU9UenJNUk5VZUw5d1VRUFRPVzF3NHVlU0l5ay9HVFJC?= =?utf-8?B?a3d2YXVUNm0yK3EyVGNQYjMvYWxiR2lHc1VyeDlTZ0lFNDlLb3U5TFYrQ1Vm?= =?utf-8?B?blFlcTlxemlCMlh6YTNUSGFXUVVTVVFEcncrOVpZOThBSUFEV24yVklwVzZt?= =?utf-8?B?cVdhNFVYOU9teTBZTzczc29LNlVaUTNLZWhVQm5Nd3dkRnlUT3c2OU53dkp2?= =?utf-8?B?Ri9xajY1RWtxN2xwSjlqRXRwOERYN1hKcmhrYmtkYWtKTCtGZ2p6SVFQUmxj?= =?utf-8?B?SGwyeVY2cC9xN0tzWmtYSW9jRHg3bVlZZXQ3Z1lqeXY4MmgwbjFJZ3JEYjZy?= =?utf-8?B?eEhMWFFLd2h5K0xLS1ErdDAwNXo3clJzeE9DUVVIRko5MTRPK05wQjh0T09Y?= =?utf-8?B?Y0lXQ1cxN2htYzBhZXJFZ2FQak80OGtYQzZQZ0ZJNVdFRzUxTEhJMEFaS2hQ?= =?utf-8?B?MXJqNzZ2bG1BbU9DTks3V2h5R0lNNUhKeWFIMkR6KzFxL2ZreWhqcFJGTEti?= =?utf-8?B?MUtvNm9UbVY5bXQ3S1VkYk52ZnhXYkxlMnlJeVV4SWdqT0R0SDRzZHNKNWJ0?= =?utf-8?B?ckdCZDBmVG5jSy9vUVBGelZ1dzFlbDkzczhKMit4eFNCSldOZnEvRDFtbXNo?= =?utf-8?B?SkgrZS9KUi9vVVVIRXRwRVVFVmZ4WU9BVDBxZEhQczFGNUFXMkdnLzc4Sm9h?= =?utf-8?B?R3UvUkpFeE1aTFpramdIMFBkR0tlU0hJTDBaSDZLeFFlb2pKK2tLWWt6T3JG?= =?utf-8?B?M1MxMHN3eTkvR1ZiMUViRDNXV04yeWhEWGdYbzNaVTdhdndDalBycW1aelRN?= =?utf-8?B?VndsVVhtZ1hpWUZJeUhzMEFtUWw4THJudzZXSVh1ckorMVR3UHl3OEwzeGQr?= =?utf-8?B?K1dsMVNYTEptN1pacjFsYVJGdlUzWjZvU0x2c2gyQ05UdUVxREpoeHlobUFK?= =?utf-8?B?RUpwZVZabSs3YzM0MGhBUVB6VGxqRVZSeHR2UWdhTWJuQUdXeURoakdxR0hq?= =?utf-8?B?TzZXMjhoNzNmWSt6RnB1NG95RVQyR2pBUjdJeUJWVlV3dkVGeVZxMHJ0SjI5?= =?utf-8?B?RGlPeGZyS1ZyYXV3M2NhR21GdWJtbWRPeU5YVlh0dXVwQS8rVEx0K253bWRR?= =?utf-8?B?Sm9FS3ZUODRNcmw4b1NRMlI1LzRCSWFKN1NiNDQ0YVlxZC9VWGdkVzR5dlJx?= =?utf-8?B?UUpXUWFZZkdZMXFHU092cjVaZkMyNmlwclZ4QzlUTUF0THZldW9WUlZGdS9p?= =?utf-8?B?TVFNcEpSNFlWa3dKL0VIeS9SbXV3RmU1dVNhNWhuTXdYTFVrZ2N0ZEJhblZ6?= =?utf-8?B?bUMxNmhrNXFzSEs2V2EzelB3Q2pnQXQvWG95Rk9YNUJpU29QUXdKaEFDMXR2?= =?utf-8?B?L1F6Yk51QXpJRmFZZ29lZ1RjcXErZkdPVkNaa1hhaTJ6ZmhEcGlqanFRZTlG?= =?utf-8?B?V1A2VUNnR0lnPT0=?= X-MS-Exchange-CrossTenant-Network-Message-Id: 36817cc5-e39b-40bc-8af4-08da0db30b96 X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Mar 2022 16:26:27.2474 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: h3r8vyN3kJ0fB0irSrrfkrlvTwjXMjodrgQ/g9Jowqn8ULm+T61iUti+XF+iPGaQIhdtsG0cX5DhFtvLh6G8Zw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR03MB3527 X-OriginatorOrg: citrix.com X-Rspamd-Queue-Id: 4KPVwY15Q7z4ll4 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=citrix.com header.s=securemail header.b=fFtIoNOT; dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=rBgBXbiw; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=reject) header.from=citrix.com; spf=pass (mx1.freebsd.org: domain of "prvs=07544547a=roger.pau@citrix.com" designates 216.71.145.153 as permitted sender) smtp.mailfrom="prvs=07544547a=roger.pau@citrix.com" X-Spamd-Result: default: False [-5.64 / 15.00]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+exists:216.71.145.153.spf.hc3370-68.iphmx.com]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; RCVD_IN_DNSWL_MED(-0.20)[216.71.145.153:from]; DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+]; DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject]; NEURAL_HAM_SHORT(-1.00)[-0.999]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_COUNT_ZERO(0.00)[0]; MIME_TRACE(0.00)[0:+]; R_MIXED_CHARSET(0.56)[subject]; ASN(0.00)[asn:16417, ipnet:216.71.145.0/24, country:US]; FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=07544547a=roger.pau@citrix.com]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; FORGED_SENDER_VERP_SRS(0.00)[]; DWL_DNSWL_LOW(-1.00)[citrix.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[freebsd-xen]; MID_RHS_NOT_FQDN(0.50)[] X-ThisMailContainsUnwantedMimeParts: N On Thu, Mar 24, 2022 at 06:01:24PM +0200, Ze Dupsys wrote: > On 2022.03.24. 15:12, Roger Pau Monné wrote: > > On Mon, Mar 21, 2022 at 06:56:05PM +0100, Roger Pau Monné wrote: > > > On Mon, Mar 21, 2022 at 05:35:15PM +0100, Roger Pau Monné wrote: > > > > On Mon, Mar 21, 2022 at 04:07:48PM +0200, Ze Dupsys wrote: > > > > > On 2022.03.21. 13:14, Roger Pau Monné wrote: > > > > > > I think the problem is not likely with the xenstore implementation > > > > > > (ie: xs_talkv) but rather a race with how the FreeBSD kernel detects > > > > > > and manages addition and removal of devices that hang off xenbus. > > > > > > > > > > > > I'm afraid there's too much data below for me to parse it. > > > > > > > > > > Understood. Sounds more tricky than i thought. What could i do to make data > > > > > more useful? > > > > > > > > I have another patch for you to try. This will make the system a bit > > > > chatty, let's see what you get. > > > > > > Forgot to mention: when testing the patch attached to the previous > > > email there's no need to push the system until you get a panic. Just > > > detecting when you have stale xbbd entries in sysctl would be enough, > > > or alternatively when you start to see entries in the output of > > > `xenstore-ls -fp` like: > > > > > > /local/domain/0/backend/vbd/XX/XXXXX = "" (n0) > > > /local/domain/0/backend/vbd/XX/XXXXX/feature-barrier = "1" (n0) > > > /local/domain/0/backend/vbd/XX/XXXXX/feature-flush-cache = "1" (n0) > > > /local/domain/0/backend/vbd/XX/XXXXX/max-ring-page-order = "5" (n0) > > > > > > Note the lack of a '/local/domain/0/backend/vbd/XX/XXXXX/state' node. > > > > > > At that point I would request that you attach the output of > > > `xenstore-ls -fp` together with the full serial log since the system > > > booted. > > > > > > You might not need a lot of iterations to trigger that state. > > > > Hello, > > > > Sorry to pester, but do you have any update on this? > > > > I'm quite sure there are races with xenbus device attach/detach, and > > the earlier we can get this sorted out the better. > > Hello, > > Yes, i agree. > > Sorry that i could not write sooner, the logs i have gathered so far does > not seem to be helpful, garbage before panic (i'll attach anyways). I did > run few tests, 2 of them did panic kernel, so i could not gather > `xenstore-ls -fp` output. > > While testing in all cases i did not notice any sysctl variable leaks, all > logged sysctls seemed to be normal. Without patch system did not panic at > first sysctl variable leak, so maybe the given patch fixed sysctl variable > leaks, but it did not solve panic problem for sure. > > I am thinking to rewrite testing scripts somehow so that information about > system is gathered in a better way. > > Would getting rid of ZFS help debugging problem? I'll try to decrease HDD > count for VMs as well, so that there is less noise in logs. Hm, TBH I'm not sure what the problem is. I've guessed it was some kind of leak due to stale backends not being properly cleaned, but it might be something else. This seems to be a fairly common trace for your panics: #0 0xffffffff80c74605 at kdb_backtrace+0x65 #1 0xffffffff80c26611 at vpanic+0x181 #2 0xffffffff80c26483 at panic+0x43 #3 0xffffffff810c1b97 at trap+0xba7 #4 0xffffffff810c1bef at trap+0xbff #5 0xffffffff810c1243 at trap+0x253 #6 0xffffffff81098c58 at calltrap+0x8 #7 0xffffffff80c7f251 at rman_is_region_manager+0x241 #8 0xffffffff80c36e71 at sbuf_new_for_sysctl+0x101 #9 0xffffffff80c362bc at kernel_sysctl+0x3ec #10 0xffffffff80c36933 at userland_sysctl+0x173 #11 0xffffffff80c3677f at sys___sysctl+0x5f #12 0xffffffff810c249c at amd64_syscall+0x10c #13 0xffffffff8109956b at Xfast_syscall+0xfb Could you give me the output of executing the following on dom0: $ addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c7f251 $ addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c36e71 $ addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c362bc $ addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c36933 $ addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80c3677f That would give us a more accurate trace. Thanks, Roger.