From nobody Sun Mar 27 09:13:42 2022 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 370851A436A5 for ; Sun, 27 Mar 2022 09:13:59 +0000 (UTC) (envelope-from prvs=0786bbb88=roger.pau@citrix.com) Received: from esa3.hc3370-68.iphmx.com (esa3.hc3370-68.iphmx.com [216.71.145.155]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KR99y0Y9Wz3Bwf for ; Sun, 27 Mar 2022 09:13:57 +0000 (UTC) (envelope-from prvs=0786bbb88=roger.pau@citrix.com) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1648372438; h=date:from:to:cc:subject:message-id:references: content-transfer-encoding:in-reply-to:mime-version; bh=sDgDS3d4+EnV4ntO5BN8ciJTN3sU6GONHafQzRWP1ck=; b=cHWaYBQ25FzVsEQV0ugZ98kBxTsxQ4gjHHx/nAR7yKPVk/J5xckHAdZS dfrnpkkQm/5g2L8piny92JRE4r6gGzEpr4FwknF28J+3JlEQEC2XkqPvX 8dnXKueAfWTjN8eT8VHZWhqURXthRUL6twRZXCoP5n6GT4KidBVzGXy68 s=; X-SBRS: 5.1 X-MesageID: 67263878 X-Ironport-Server: esa3.hc3370-68.iphmx.com X-Remote-IP: 162.221.156.83 X-Policy: $RELAYED IronPort-Data: A9a23:4qocl6MtxUU9JTXvrR2yl8FynXyQoLVcMsEvi/4bfWQNrUpzhGFSx 2tLXT2GPv/eYWLzfdsiaYTn9kMF75WGy9UxGQto+SlhQUwRpJueD7x1DKtR0wB+jCHnZBg6h ynLQoCYdKjYdleF+lH1dOKJQUBUjclkfJKlYAL/En03FFcMpBsJ00o5wbZi2NAw2rBVPivW0 T/Mi5yHULOa82Yc3lI8s8pvfzs24ZweEBtB1rAPTagjUG32zhH5P7pGTU2FFFPqQ5E8IwKPb 72rIIdVXI/u10xF5tuNyt4Xe6CRK1LYFVDmZnF+A8BOjvXez8CbP2lS2Pc0MC9qZzu1c99Z1 PsTmoCcWz8SAqDtv+9CCQBfOR0vMvgTkFPHCSDXXc27ykTHdz3nwul0DVFwNoodkgp1KTgQr 7pCcmlLN03dwbLtqF64YrAEasALNs7kMZlZonh95TrYEewnUdbIRKCiCdpwgmhq2JseQqa2i 8wxVX1dZROdfgRzYWgdApZuusi3gEPRSmgNwL6SjfVuuDWCpOBr65DnLdrPfcOiQ8tflU+c4 GXc8AzRDAsXOdaExX+V/3SEiPXCkCngVMQVDrLQ3uxnjECL3X0IIBITXFq/5/K+jyaDt8l3c hJOvHB09O5rqRLtHoKVswCETGCsojBCdMZ8AdUB1Ays7aGLuR+eLFRVQWsUADA5j/MeSTsv3 16PutrmAz1zrbGYIU6gGqeoQSCaYnZMczJbDcMQZU5cuoS4/tlv5v7aZow7eJNZmOEZDt0ZL 9qiiCElz4segscQv0lQ1QCW2mn8znQlo+Nc2+k2Yo5Hxl4jDGJGT9bxgbQ+0RqmBN3AJrVml CJY8/VyFMhUUfmweNWlGY3h5o2B6fefKyH7ilVyBZQn/DnF0yf9IdALumkheB8yaZxsldrVj Kn74lo5CHh7ZibCUEOKS9jpV5RCIVbIS7wJqcw4nvIRO8MsJWdrDQllZFKK3nCFraTfufpXB HtvSu71VSxyIf0+lFKeHr5BuZd2lnFW7T6CHvjTkkX4uYdykVbIEN/pxnPVNbtnhE5FyS2Im +ti2zyilkwHCbakM3CNmWPRRHhTRUUG6VnNg5U/XsaIIxZ8GXFnDPnUwLg7fJdikbgTneDNl kxRkGcBoLYjrRUr8Tm3V00= IronPort-HdrOrdr: A9a23:JNsrOK9qY9tWxR4HqUduk+E6db1zdoMgy1knxilNoENuHfBwxv rDoB1E73LJYVYqOU3Jmbi7Sc69qFfnhORICO4qTMqftWjdyRCVxeRZg7cKrAeQeREWmtQtsJ uINpIOdOEYbmIK/PoSgjPIaurIqePvmMvD5Za8854ud3ATV0gJ1XYGNu/xKDwReOApP+tcKH LKjfA32AZINE5nJviTNz0gZazuttfLnJXpbVovAAMm0hCHiXeN5KThGxaV8x8CW3cXqI1Su1 Ttokjc3OGOovu7whjT2yv66IlXosLozp9mCNaXgsYYBz3wgkKDZZhnWZeFoDcpydvfo2oCoZ 3pmVMNLs5z43TeciWcpgbs4RDp1HIU53rr2Taj8AzeiP28YAh/J9tKhIpffBecwVEnpstA3K VC2H/cn4ZLDDvb9R6NqOTgZlVPrA6ZsHAimekcgzh0So0FcoJcqoQZ4Qd8DIoAJiTn84oqed MeQP003MwmMG9yUkqp/lWGmLeXLzcO91a9MwU/U/WuonZrdCsT9Tpb+CQd9k1wgK7VBaM0ot gsCZ4Y542mfvVmHZ6VO91xM/dfKla9Ny4kY1jiaGgOKsk8SgfwQtjMkfEI2N0= X-IronPort-AV: E=Sophos;i="5.90,214,1643691600"; d="scan'208";a="67263878" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=CynrBxGIDrAbVvIXSdHzrBUvVPtk6IFBg4oAiKNIzwn2zNFa9QRZ2sU3VNTyENy5rfPRnrhKYOcmREzXvNQuioX8Yle1fVALWfk40PryBVWoyLPN02iz1m1TVkeXKoFR5Sdhh74c6BwzmlzFLmsIGry3jXgdXm7OdCEDj+/rDRhRT1yIDTDte8Hjev/3tbvONEdInuf6u0Uk/pX02Va6Jj8ReVRNCaLVRSDKYuqykn8mPNRH3ah5yrINGT0ASsgxZ42H6En6Rum8A0yHn6igRlWIbmofvY0bLarPP6s+U3WYXWN349e13YALIv/YQS/qnP5o4bAcAl+N92n2Xx5KaQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=phK1VtnyV6Q2W2ongc6zZ1VKNl6MB4ofLvHZ+OMp2J0=; b=i8Xf/NcVNQUd1b344OgvCPMgj2FcgWupdJd+RRn7VCes3Y0ON5pfmb9QQ5cZnVKgIw0H6SJ1rJ/o/PEi7bB63Su8kSyyCZ2kCF7T7kxI+eu4/W5SpJJP+lTQZA6PDJ2+Jizlp/R5f3hAEOH15zGS6DudmsONoBQU4kABboxLsaUSflAOys+cm2Dzy0Oc7vP460iey470ijtVqIkG5zDnlrfJ0CoSdNeNNmk0FmuGFra9qcnwzHh9DSNSqFSBk+DSVaZWwY6gSoHbOEp4F2ke/3nJBSpdjbfrZz+VOS+xe12I0mqe9+mbMStKUKQ7jUWqzpUO1sn9jYWkb7JntshXSQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com; dkim=pass header.d=citrix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=phK1VtnyV6Q2W2ongc6zZ1VKNl6MB4ofLvHZ+OMp2J0=; b=Xw7RhY0aGpTsHXF3Eg5A14WyKJD3Jb8HHL3dkWSA2YNnm8mmPebAeR+Sh91iSPOa1cIMrp96uvWISMkjA460xnysVnjGSY7koh3I7gQ+ze+zgcgMa45eWxbbsIPUdjE+KZFiFInJXcIO/pElLaMeWUT4ZLcZcRboeuqlSAZZR0g= Date: Sun, 27 Mar 2022 11:13:42 +0200 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Ze Dupsys CC: , Subject: Re: ZFS + FreeBSD XEN dom0 panic Message-ID: References: <088c8222-063a-1db5-da83-a5a0168d66c6@gmail.com> <639f7ce0-8a07-884c-c1cf-8257b9f3d9e8@gmail.com> <4da2302b-0745-ea1d-c868-5a8a5fc66b18@gmail.com> <48b74c39-abb3-0a3e-91a8-b5ab1e1223ce@gmail.com> Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <48b74c39-abb3-0a3e-91a8-b5ab1e1223ce@gmail.com> X-ClientProxiedBy: MR2P264CA0070.FRAP264.PROD.OUTLOOK.COM (2603:10a6:500:31::34) To DS7PR03MB5608.namprd03.prod.outlook.com (2603:10b6:5:2c9::18) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: e974ca05-fe96-47d3-7904-08da0fd219e9 X-MS-TrafficTypeDiagnostic: MN2PR03MB4829:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: R2Q7ZKnhae89zKCqCwKWWC7AMSOr6kzflfOgU51xrykLMbxCPzndQiDrRu7Z4ZoQS4zIWpDt3CrHYPpwmtRSOvvb92GPvH7xhPuMpRh49OhJR6xZI9jcQKwWsYsMu0Q1F/Bx0rwaTg2mfXKLd9rumTntpBMu1++XuvJi51OYGeKxUEXIvhAKtRtEQ84sUaLd0oCnq3sF8zYhvWultC8IBloxlv7PWDzH/JSzTYkUiUQT/uVdjiKmu7q8rnMiC4PEeDTn1T6NKinFfkOkYfPf2LXblOzt6qevpi+GBM2oNh/JFimogPvvkyhw7nqDptwcfVfGN+eWrXNRMIwwOuUyuNwVnC/yeT+DfY8dNcorj8bEv2Muv3LRjUMglXbczSbi+To5U5o2o+eaN3MIhRCqwiDslhSZJA4HxzgVbYJc53BgACY8rBqa6xHkxcHexExKsXYUgUh4EjSuNAO5XKp7rT4kzCzksUK+TfczTWTHBDwbSyA1YcDRlpGDoSYMiQCjSCMnTPuZQXtC4iAUipeQnJDwrKF073H2dR3yxe7gvUNKdNzWQhah2EmwduZCZDPGywEHAo5+h8B79uOrpGw/0UTo9b+zPkd4qHWY5cmFB9u3MqUVvx4l7oDbVLFilia3tqLheKGogsilYAkF8ZRBeA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(7916004)(4636009)(366004)(83380400001)(6486002)(508600001)(186003)(2906002)(33716001)(8936002)(86362001)(6512007)(6506007)(30864003)(6916009)(82960400001)(316002)(9686003)(38100700002)(85182001)(6666004)(26005)(4326008)(66556008)(66476007)(5660300002)(66946007)(8676002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bnJoK2JtT09wVmdqT3B2QktYSFFaQTQ3bXhoM2hrdlBGanBOZktXSFpCZTZU?= =?utf-8?B?Qld0YzB5ZmNuUlNOMUFHVXJ3VG1vcVBDaDMzekVNY0puelNYSjRSZlkySWJ1?= =?utf-8?B?QnplTTE4c0FRR0pjZWpHZ0dUbVcwN29scDJIR3lsSmcySU5nM0F1WmFKTjgw?= =?utf-8?B?NWdOZzlKNVJ4azg5K1FBYzBqaWdSayswTGFodEZlZGRodi9pMHhoS3V2WVV5?= =?utf-8?B?VWxDd0YzajhlTU45NlBBV0Q5MGFtVjlabTdOYWtta3ViSjQwNXVZelF0YlVv?= =?utf-8?B?NWNPTC9zMy9LNkhZZEp1NXBRaTN2UzlwZ1BPdlk0dmRJRExFS2JLYUxhbmdH?= =?utf-8?B?cU8zKzc3UjgwTW13UkZxOXRzQmlGR21HelJMc0ZZV0g0SWJjYmx1OENFdHNk?= =?utf-8?B?dlNNRklvMXpoMlJCUG1SekFJcEUzSDRscFN4cDkySWdzS1pOeVd3L0dJYVEv?= =?utf-8?B?VEdIRUVmU1EzaHVmREFqTGV3Z0g0bllIU1FwbzFNMUZpak5QOVBCWVc4QU03?= =?utf-8?B?OE1rSkttaURhTE16Z1FWa0Q4R3k0V2o2WkZaaXdEMXJpbmlNNzJoZ1dkSCtG?= =?utf-8?B?NHBIRXZQYkJWZkt4cHEvaGRCSkF3SVJHeXhSUjYxVHJjdjZLQ1dVUkZBazhI?= =?utf-8?B?QlBpQng2VmJsMWUzb1dsdGt0OUkrQUVVMkRwM1RFTTdpZDJDWExRYzI2N1VO?= =?utf-8?B?bVhkczFzZmVKbmppTDVKR2htcU9BWTk3TUdvTnU3ZzZJWjUxeDh6d3NLNjFR?= =?utf-8?B?Uk5GdVorVjFLMThpK0oxaWxSVkMwdDFtS3FRWHlOaEo1SHNWMXpSVkpjUzBH?= =?utf-8?B?THdaY3BBMVRKd2YvWGp0VTZOMnRpTVpERmxRZjQ1d2pja2lZejhVSmsyV2hQ?= =?utf-8?B?WHpwWlR5Q01zNGtubmhjTFNyVTdHZGQ4bTNFMTBXQ3g2bVl4dW1tdzNhZTVJ?= =?utf-8?B?UWF1OWlVcXZFYnQvSVM0c1R3ditESi96b1k2dkFpVW1GY1RLdGF0TXJucm5l?= =?utf-8?B?U1BoWWJVWlg5RzBRN2d2cG02WDFxYXFKanNSNTVEOE4wd0FQU2pBYklyNk1h?= =?utf-8?B?aUdVd3JOK3lBUWh5b0U0MWxPVmc2L1lob01yNVlPOU5XVjB2aTJUcjZNcU5Y?= =?utf-8?B?QjZtZ0s1WDYyN3F5SzV4UXMyV1RCNCtPb1pmclVlNjl4Q0FuSVNBV0JhMUNR?= =?utf-8?B?dldieGpVSXFTWmdTVlI5bXROZzhlQVhQUmlDU0szUWEzY2p1Z2dpZ21XRG5H?= =?utf-8?B?OTRuYnFIWU95QnlrUEJ1clJJcTNpTG5jeDRad2RIemZiaDZlOWdzR29GMFF2?= =?utf-8?B?SmtFeE9vaXU4eStjY0JvQVAvais5aEVtdXFzU24vQTg5alRkbEdMSUwrZGJn?= =?utf-8?B?dktMQWw5bEMrTHNFbXlxVmgrYXU5YWQyUXBBNnhORFFENzhuVTdEdm16UFRy?= =?utf-8?B?R1l4YzIzVUxYbnR3akI4NW43YzdTcFpQMHhNSXFUWkhMWkZlcGlyRTFIQ092?= =?utf-8?B?WWl5MVpxZnNjQ2FPZFdEbnBXRG1nbEhUM1ZFamxCNTd5T0ltM2dENlZBR3lp?= =?utf-8?B?aUZOL1FHVVQzV240QlM1cEl3TCtXMHNEdVdTNW5lU2xWSWw0SWJFUzJ4TTNM?= =?utf-8?B?S3hsNUM5WGQvMEVhRHJ5cUlGUVRsc2tzSGtyQlkrSVg2TnJtaEZhSDA5bnJB?= =?utf-8?B?R2Z5RnRGL1pOa1I3dGlvOXIxbW80eEl2dkF0Mkg5ZXZEYzZrekIxa3BKaFhL?= =?utf-8?B?c1l0V2lQWWgxdjlnVFlXd0VUOTFsMEltdzRoVEJoNFFSRjA1NnYzNlFKNkEv?= =?utf-8?B?U29rQUpiMWVtWmQ2Q20zcEZmR0EyZjNzeVNKMUFmTWxwQ05kcXJwTjJaeWNl?= =?utf-8?B?UldJakZhdzY5cjBKK0dPdjhQUHY0Vzh3eUJDRGx3MlNFSHRYTnAxWGk0TE9L?= =?utf-8?B?SkJzb1QwT2pTbW5oYnpyMXd2Z2c4UGkwNlFWL1ZKYm9iM21lY2tzYnZCUUJ3?= =?utf-8?B?ZGx1UmpCNnRQcmg3SVF4WDYvRlNUVFFCYno3TzltemwwSG5OMjA5N3BpZjhY?= =?utf-8?B?d01uOEZqdEtKWXhhZGFYb0ZHeTQvMndrT1d0cWlNVXEvMDNEbVZhaysrNHNw?= =?utf-8?B?VUorR3BabDhqZXpmNDZrb2hhcDBiRjNoVzJiMFRkQlhJcFJPMlBjWUZlVFk1?= =?utf-8?B?THRlYVRUMVVmRWhLc1YvR0FqZ0dvdWl5T0taWnRDeE1rZFdhR2taSjRJcjJi?= =?utf-8?B?WGN6ZlNxUy8zSjhHSldLRFVyVnp2Y2lVRlQ5aEc0ZWd6cllOTnN6VVYyMG9u?= =?utf-8?B?OHR0bzJBeW1lOGwyd2dWbEhEQ1NDMDdqdk43SzlpT05HSW1mc0tVQT09?= X-MS-Exchange-CrossTenant-Network-Message-Id: e974ca05-fe96-47d3-7904-08da0fd219e9 X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2022 09:13:48.1750 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: B4pncfJ0dgMRzPUUmv/83Sf0xp60rIyiddiyG+flEtvI2c2gBipL+Hz5lYulKEUA0luDViENgaT7lA0Iva+qRA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR03MB4829 X-OriginatorOrg: citrix.com X-Rspamd-Queue-Id: 4KR99y0Y9Wz3Bwf X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=citrix.com header.s=securemail header.b=cHWaYBQ2; dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=Xw7RhY0a; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=reject) header.from=citrix.com; spf=pass (mx1.freebsd.org: domain of "prvs=0786bbb88=roger.pau@citrix.com" designates 216.71.145.155 as permitted sender) smtp.mailfrom="prvs=0786bbb88=roger.pau@citrix.com" X-Spamd-Result: default: False [-5.61 / 15.00]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+exists:216.71.145.155.spf.hc3370-68.iphmx.com]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; RCVD_IN_DNSWL_MED(-0.20)[216.71.145.155:from]; DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+]; DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject]; NEURAL_HAM_SHORT(-0.96)[-0.963]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_COUNT_ZERO(0.00)[0]; MIME_TRACE(0.00)[0:+]; R_MIXED_CHARSET(0.56)[subject]; ASN(0.00)[asn:16417, ipnet:216.71.145.0/24, country:US]; FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=0786bbb88=roger.pau@citrix.com]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; FORGED_SENDER_VERP_SRS(0.00)[]; DWL_DNSWL_LOW(-1.00)[citrix.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[freebsd-xen]; MID_RHS_NOT_FQDN(0.50)[] X-ThisMailContainsUnwantedMimeParts: N On Sun, Mar 27, 2022 at 12:38:00AM +0200, Ze Dupsys wrote: > On 2022.03.26. 16:38, Roger Pau Monné wrote: > > .. > > It's weird, because here you get a page fault, but there are also > > traces with: > > .. > > general protection fault while in kernel mode > > .. > > That show a general protection fault instead of a page fault. > > Yes indeed, i had not noticed this. Grepped across 34 stored panic log > files, i see that 28 are page fault, 4 are general protection fault, 2 > other. I though maybe RAM size influences this, but page faults have 2G, 4G, > 6G, 8G Dom0, general protection faults have 2G, 4G, 8G. > > I have no idea what triggers what, since stress tests and command line args > are more or less the same. Builds are different with patches, some debug > info, etc. Almost all panic traces have "rman_is_region_manager" in mid, > actually looking all of them together seemed interesting. I'll attach unique > panic traces, since some included snprintf, kvprintf as well, maybe helpful. > Unfortunately i do not know which version and what patches were applied. > > > > I've also noticed it seems to always be 'devmatch' the process that > > triggers the panic. > > Yes, it seems to be the case most of the time. There are 3 cases when > process is "xbbd* taskq". 2 cases with 2G RAM, 1 with 6G. > > > > I've been able to get a better trace with gdb and your debug symbols, > > and this is: > > > > (gdb) info line *0xffffffff80c6a2b2 > > Line 1386 of "/usr/src/sys/kern/subr_bus.c" starts at address > 0xffffffff80c6a2b2 > > and ends at 0xffffffff80c6a2b6 . > > (gdb) info line *0xffffffff80c86ed1 > > Line 1052 of "/usr/src/sys/kern/subr_rman.c" starts at address > 0xffffffff80c86ecc > > and ends at 0xffffffff80c86ed5 . > > This is a nice find! > > > > I'm trying to figure out how the device could be removed or > > disconnected from the rman. I will try to create a patch to catch the > > device that leaves rman regions when destroyed/removed. > > Okay, i'll apply when it will be possible. > > I did run xen-debug on system with applied blkback.patch as you sent in next > message to this. > > System had panic with new trace: > Fatal trap 12: page fault while in kernel mode > cpuid = 2; apic id = 04 > fault virtual address = 0xa4 > fault code = supervisor read data, page not present > instruction pointer = 0x20:0xffffffff80c90ed0 > stack pointer = 0x28:0xfffffe0051927ab0 > frame pointer = 0x28:0xfffffe0051927ad0 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 16 (xenwatch) > trap number = 12 > panic: page fault > cpuid = 1 > time = 1648331592 > KDB: stack backtrace: > #0 0xffffffff80c7c275 at kdb_backtrace+0x65 > #1 0xffffffff80c2e2d1 at vpanic+0x181 > #2 0xffffffff80c2e143 at panic+0x43 > #3 0xffffffff810c8b97 at trap+0xba7 > #4 0xffffffff810c8bef at trap+0xbff > #5 0xffffffff810c8243 at trap+0x253 > #6 0xffffffff810a0838 at calltrap+0x8 > #7 0xffffffff80a98515 at xbd_instance_create+0x7895 > #8 0xffffffff80a98462 at xbd_instance_create+0x77e2 > #9 0xffffffff80a9619b at xbd_instance_create+0x551b > #10 0xffffffff80f95c54 at xenbusb_localend_changed+0x7c4 > #11 0xffffffff80ab0ef4 at xs_unlock+0x704 > #12 0xffffffff80beaede at fork_exit+0x7e > #13 0xffffffff810a18ae at fork_trampoline+0xe > > cat /tmp/panic.log| sed -Ee 's/^#[0-9]* //' -e 's/ .*//' | xargs addr2line > -e /usr/lib/debug/boot/kernel/kernel.debug > > /usr/src/sys/kern/subr_kdb.c:443 > /usr/src/sys/kern/kern_shutdown.c:0 > /usr/src/sys/kern/kern_shutdown.c:844 > /usr/src/sys/amd64/amd64/trap.c:944 > /usr/src/sys/amd64/amd64/trap.c:0 > /usr/src/sys/amd64/amd64/trap.c:0 > /usr/src/sys/amd64/amd64/exception.S:292 > /usr/src/sys/dev/xen/blkback/blkback.c:2789 > /usr/src/sys/dev/xen/blkback/blkback.c:3431 > /usr/src/sys/dev/xen/blkback/blkback.c:3912 > /usr/src/sys/xen/xenbus/xenbusb_back.c:238 > /usr/src/sys/dev/xen/xenstore/xenstore.c:1007 > /usr/src/sys/kern/kern_fork.c:1099 > /usr/src/sys/amd64/amd64/exception.S:1091 Thanks, unfortunately that patch was incomplete. I have an updated version that I think is better now, and I've slightly tested it (creating and destroying a domain with it doesn't seem to crash). Appended patch at the end of the message. > > Full serial log in attachment. > > Thanks. > ==== COUNT: 1 > Fatal trap 9: general protection fault while in kernel mode > cpuid = 0; apic id = 00 > instruction pointer = 0x20:0xffffffff80c45892 > stack pointer = 0x28:0xfffffe00d2d2b930 > frame pointer = 0x28:0xfffffe00d2d2b930 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 10984 (devmatch) > trap number = 9 > panic: general protection fault > cpuid = 0 > time = 1646305680 > KDB: stack backtrace: > #0 0xffffffff80c57525 at kdb_backtrace+0x65 > #1 0xffffffff80c09f01 at vpanic+0x181 > #2 0xffffffff80c09d73 at panic+0x43 > #3 0xffffffff8108b1a7 at trap+0xbc7 > #4 0xffffffff8108a66e at trap+0x8e > #5 0xffffffff81061b18 at calltrap+0x8 > #6 0xffffffff80c62011 at rman_is_region_manager+0x241 > #7 0xffffffff80c1a051 at sbuf_new_for_sysctl+0x101 > #8 0xffffffff80c1949c at kernel_sysctl+0x43c > #9 0xffffffff80c19b13 at userland_sysctl+0x173 > #10 0xffffffff80c1995f at sys___sysctl+0x5f > #11 0xffffffff8108baac at amd64_syscall+0x10c > #12 0xffffffff8106243e at Xfast_syscall+0xfe > Uptime: 1h15m46s > > > ==== COUNT: 3 > Fatal trap 9: general protection fault while in kernel mode > cpuid = 0; apic id = 00 > instruction pointer = 0x20:0xffffffff80d0728f > stack pointer = 0x28:0xfffffe00a17ea790 > frame pointer = 0x28:0xfffffe00a17ea790 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 2785 (devmatch) > trap number = 9 > panic: general protection fault > cpuid = 1 > time = 1646419029 > KDB: stack backtrace: > #0 0xffffffff80c57525 at kdb_backtrace+0x65 > #1 0xffffffff80c09f01 at vpanic+0x181 > #2 0xffffffff80c09d73 at panic+0x43 > #3 0xffffffff8108b1a7 at trap+0xbc7 > #4 0xffffffff8108a66e at trap+0x8e > #5 0xffffffff81061b18 at calltrap+0x8 > #6 0xffffffff80c5da17 at kvprintf+0x1007 > #7 0xffffffff80c5e719 at snprintf+0x59 > #8 0xffffffff80c6204b at rman_is_region_manager+0x27b > #9 0xffffffff80c1a051 at sbuf_new_for_sysctl+0x101 > #10 0xffffffff80c1949c at kernel_sysctl+0x43c > #11 0xffffffff80c19b13 at userland_sysctl+0x173 > #12 0xffffffff80c1995f at sys___sysctl+0x5f > #13 0xffffffff8108baac at amd64_syscall+0x10c > #14 0xffffffff8106243e at Xfast_syscall+0xfe > Unique on "current process" and trace fingerprint #0-#*. > > ==== COUNT: 23 > Fatal trap 12: page fault while in kernel mode > cpuid = 0; apic id = 00 > fault virtual address = 0x22710028 > fault code = supervisor read data, page not present > instruction pointer = 0x20:0xffffffff80c45892 > stack pointer = 0x28:0xfffffe0096600930 > frame pointer = 0x28:0xfffffe0096600930 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 1496 (devmatch) > trap number = 12 > panic: page fault > cpuid = 0 > time = 1646123791 > KDB: stack backtrace: > #0 0xffffffff80c57525 at kdb_backtrace+0x65 > #1 0xffffffff80c09f01 at vpanic+0x181 > #2 0xffffffff80c09d73 at panic+0x43 > #3 0xffffffff8108b1a7 at trap+0xbc7 > #4 0xffffffff8108b1ff at trap+0xc1f > #5 0xffffffff8108a85d at trap+0x27d > #6 0xffffffff81061b18 at calltrap+0x8 > #7 0xffffffff80c62011 at rman_is_region_manager+0x241 > #8 0xffffffff80c1a051 at sbuf_new_for_sysctl+0x101 > #9 0xffffffff80c1949c at kernel_sysctl+0x43c > #10 0xffffffff80c19b13 at userland_sysctl+0x173 > #11 0xffffffff80c1995f at sys___sysctl+0x5f > #12 0xffffffff8108baac at amd64_syscall+0x10c > #13 0xffffffff8106243e at Xfast_syscall+0xfe > > > > ==== COUNT: 2 > current process = 20284 (devmatch) > trap number = 12 > panic: page fault > cpuid = 3 > time = 1647247618 > KDB: stack backtrace: > #0 0xffffffff80c7c615 at kdb_backtrace+0x65 > #1 0xffffffff80c2e621 at vpanic+0x181 > #2 0xffffffff80c2e493 at panic+0x43 > #3 0xffffffff810c8b97 at trap+0xba7 > #4 0xffffffff810c8bef at trap+0xbff > #5 0xffffffff810c8243 at trap+0x253 > #6 0xffffffff810a09d8 at calltrap+0x8 > #7 0xffffffff80c82c77 at kvprintf+0x1007 > #8 0xffffffff80c83a09 at snprintf+0x59 > #9 0xffffffff80c8729b at rman_is_region_manager+0x27b > #10 0xffffffff80c3ee81 at sbuf_new_for_sysctl+0x101 > #11 0xffffffff80c3e2cc at kernel_sysctl+0x3ec > #12 0xffffffff80c3e943 at userland_sysctl+0x173 > #13 0xffffffff80c3e78f at sys___sysctl+0x5f > #14 0xffffffff810c949c at amd64_syscall+0x10c > #15 0xffffffff810a12eb at Xfast_syscall+0xfb Thanks, those all seem to be related to a device being removed without cleaning it's rman regions properly. So far I've spotted an issue in blkback in this regard, but I wouldn't discard other issues in either blkback or netback. Let's see if the updated blkback patch makes a difference now. > ==== COUNT: 2 > Fatal trap 12: page fault while in kernel mode > cpuid = 1; apic id = 02 > fault virtual address = 0x68 > fault code = supervisor read data, page not present > instruction pointer = 0x20:0xffffffff824a599d > stack pointer = 0x28:0xfffffe00b1e27910 > frame pointer = 0x28:0xfffffe00b1e279b0 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 0 (xbbd7 taskq) > trap number = 12 > panic: page fault > cpuid = 1 > time = 1646122723 > KDB: stack backtrace: > #0 0xffffffff80c57525 at kdb_backtrace+0x65 > #1 0xffffffff80c09f01 at vpanic+0x181 > #2 0xffffffff80c09d73 at panic+0x43 > #3 0xffffffff8108b1a7 at trap+0xbc7 > #4 0xffffffff8108b1ff at trap+0xc1f > #5 0xffffffff8108a85d at trap+0x27d > #6 0xffffffff81061b18 at calltrap+0x8 > #7 0xffffffff8248935a at dmu_read+0x2a > #8 0xffffffff82456a3a at zvol_geom_bio_strategy+0x2aa > #9 0xffffffff80a7f214 at xbd_instance_create+0xa394 > #10 0xffffffff80a7b1ea at xbd_instance_create+0x636a > #11 0xffffffff80c6b1c1 at taskqueue_run+0x2a1 > #12 0xffffffff80c6c4dc at taskqueue_thread_loop+0xac > #13 0xffffffff80bc7e3e at fork_exit+0x7e > #14 0xffffffff81062b9e at fork_trampoline+0xe > > > ==== COUNT: 1 > Fatal trap 12: page fault while in kernel mode > cpuid = 1; apic id = 02 > fault virtual address = 0x148 > fault code = supervisor read data, page not present > instruction pointer = 0x20:0xffffffff8248cef4 > stack pointer = 0x28:0xfffffe009941d9a0 > frame pointer = 0x28:0xfffffe009941d9a0 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 0 (xbbd1 taskq) > trap number = 12 > panic: page fault > cpuid = 1 > time = 1646129773 > KDB: stack backtrace: > #0 0xffffffff80c57525 at kdb_backtrace+0x65 > #1 0xffffffff80c09f01 at vpanic+0x181 > #2 0xffffffff80c09d73 at panic+0x43 > #3 0xffffffff8108b1a7 at trap+0xbc7 > #4 0xffffffff8108b1ff at trap+0xc1f > #5 0xffffffff8108a85d at trap+0x27d > #6 0xffffffff81061b18 at calltrap+0x8 > #7 0xffffffff825cb76e at zil_open+0xe > #8 0xffffffff82456d02 at zvol_ensure_zilog+0xb2 > #9 0xffffffff82456818 at zvol_geom_bio_strategy+0x88 > #10 0xffffffff80a7f214 at xbd_instance_create+0xa394 > #11 0xffffffff80a7b1ea at xbd_instance_create+0x636a > #12 0xffffffff80c6b1c1 at taskqueue_run+0x2a1 > #13 0xffffffff80c6c4dc at taskqueue_thread_loop+0xac > #14 0xffffffff80bc7e3e at fork_exit+0x7e > #15 0xffffffff81062b9e at fork_trampoline+0xe Hm, those last ones are in ZFS code, can you try to get the line numbers for those? Maybe it's blkback providing bad data to the disk open functions. Since you are doing so much testing, it might make sense for you to use a debug FreeBSD kernel rather than a production one (one with WITNESS and INVARIANTS enabled). Thanks, Roger. ---8<--- diff --git a/sys/dev/xen/blkback/blkback.c b/sys/dev/xen/blkback/blkback.c index 33414295bf5e..4007a93a54c7 100644 --- a/sys/dev/xen/blkback/blkback.c +++ b/sys/dev/xen/blkback/blkback.c @@ -2774,19 +2774,12 @@ xbb_free_communication_mem(struct xbb_softc *xbb) static int xbb_disconnect(struct xbb_softc *xbb) { - struct gnttab_unmap_grant_ref ops[XBB_MAX_RING_PAGES]; - struct gnttab_unmap_grant_ref *op; - u_int ring_idx; - int error; - DPRINTF("\n"); - if ((xbb->flags & XBBF_RING_CONNECTED) == 0) - return (0); - mtx_unlock(&xbb->lock); xen_intr_unbind(&xbb->xen_intr_handle); - taskqueue_drain(xbb->io_taskqueue, &xbb->io_task); + if (xbb->io_taskqueue != NULL) + taskqueue_drain(xbb->io_taskqueue, &xbb->io_task); mtx_lock(&xbb->lock); /* @@ -2796,19 +2789,28 @@ xbb_disconnect(struct xbb_softc *xbb) if (xbb->active_request_count != 0) return (EAGAIN); - for (ring_idx = 0, op = ops; - ring_idx < xbb->ring_config.ring_pages; - ring_idx++, op++) { - op->host_addr = xbb->ring_config.gnt_addr - + (ring_idx * PAGE_SIZE); - op->dev_bus_addr = xbb->ring_config.bus_addr[ring_idx]; - op->handle = xbb->ring_config.handle[ring_idx]; - } + if (xbb->flags & XBBF_RING_CONNECTED) { + struct gnttab_unmap_grant_ref ops[XBB_MAX_RING_PAGES]; + struct gnttab_unmap_grant_ref *op; + unsigned int ring_idx; + int error; + + for (ring_idx = 0, op = ops; + ring_idx < xbb->ring_config.ring_pages; + ring_idx++, op++) { + op->host_addr = xbb->ring_config.gnt_addr + + (ring_idx * PAGE_SIZE); + op->dev_bus_addr = xbb->ring_config.bus_addr[ring_idx]; + op->handle = xbb->ring_config.handle[ring_idx]; + } - error = HYPERVISOR_grant_table_op(GNTTABOP_unmap_grant_ref, ops, - xbb->ring_config.ring_pages); - if (error != 0) - panic("Grant table op failed (%d)", error); + error = HYPERVISOR_grant_table_op(GNTTABOP_unmap_grant_ref, ops, + xbb->ring_config.ring_pages); + if (error != 0) + panic("Grant table op failed (%d)", error); + + xbb->flags &= ~XBBF_RING_CONNECTED; + } xbb_free_communication_mem(xbb); @@ -2839,7 +2841,6 @@ xbb_disconnect(struct xbb_softc *xbb) xbb->request_lists = NULL; } - xbb->flags &= ~XBBF_RING_CONNECTED; return (0); }