From nobody Thu Mar 03 10:39:43 2022 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id D938719E8DE4 for ; Thu, 3 Mar 2022 10:40:04 +0000 (UTC) (envelope-from prvs=05474af58=roger.pau@citrix.com) Received: from esa2.hc3370-68.iphmx.com (esa2.hc3370-68.iphmx.com [216.71.145.153]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4K8SDM6TyTz4jsB for ; Thu, 3 Mar 2022 10:40:03 +0000 (UTC) (envelope-from prvs=05474af58=roger.pau@citrix.com) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1646304003; h=date:from:to:cc:subject:message-id:references: in-reply-to:mime-version; bh=0QyO5x2TzQ0Wl5JKLRB+cA96L2WDicG+Y4Pi+EX2GKw=; b=NEAIt31BpD349P3f26/qzY0BefaX2WVu8i8fi93yTqLLE04U+UGr4M1D 8JnnYwxfrYHp5uLFJpZzmEjBSCoAEEauKBezlvRTwgTsKKDWy1jzjVUSt Sd4ojH46TEcraQL8lwM0yOwgznoKJFkjIdxlwyfepvmeWbxXVA+YS8bN2 s=; X-SBRS: 5.1 X-MesageID: 65382780 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.156.83 X-Policy: $RELAYED IronPort-Data: A9a23:TGx1cajPcINpWhRkF3gIMUXbX161cRAKZh0ujC45NGQN5FlHY01je htvDWiHa/7YZWf3fogkPY60801Q7JbSyYNkTwBu+y9gEygb9cadCdqndUqhZCn6wu8v7a5EA 2fyTvGacajYm1eF/k/F3oDJ9CU6jefSLlbFILas1hpZHGeIcw98z0M78wIFqtQw24LhWFvW4 YqaT/D3YzdJ5RYlagr41IrbwP9flKyaVOQw5wFWiVhj5TcyplFNZH4tDfjZw0jQG+G4KtWSV efbpIxVy0uCl/sb5nFJpZ6gGqECaua60QFjERO6UYD66vRJjnRaPqrWqJPwwKqY4tmEt4kZ9 TlDiXC/YVgqMfT0icg8aQYbEiZGJJ8c9LLteFHq5KR/z2WeG5ft6/BnDUVwNowE4OdnR2pJ8 JT0KhhUMErF3bjvhuvmFK883azPL+GyVG8bkmtnwjzDS+4vXLjIQrnQ5M8e1zA17ixLNaiDP JpDNmo3BPjGSwxqP35UGa87oMOloXTwVQZClQ7LpaVitgA/yyQuieOwYbI5YOeiT9hVhEGHj mTI9W/wBlcdLtP34TSf/3+mneOKjS79cIwIGbCy7fIsh0ecrkQBBRELTUCqutG2j0e/X5RUL El8x8Y1hfFsrgrxFIC7BkDm5i7f1vIBZzZOO7I551i124b12RnaB288VwwbZNEM7eZjEFTGy WS1t9/uADVutpicRnSc6qqYoFuOBMQFEYMRTXRaFFVYurEPtKl210uSFYg7TMZZm/WoQWmY/ tyckMQpa1z/Z+Yv3r7zw13IiinESnPhHl9svVW/so5IA2pEiG+Zi26AtACzARVodt/xory9U J4swZL2AAcmV8zlqcB1aL9RdIxFHt7cWNEmvXZhHoM66xOm8GO5cIZb7VlWfRk1bJxYJ2O4O xCO4Gu9AaO/2lPwN8ebhKrrVqwXIVXIT4y5Bpg4kPIUCnSOSON31H43PhPBt4wcuEMtjbs+K f+mnTWEVh4n5VBc5GPuHY81iOZzrghnnD+7bc2rnnyPjOvFDFbIGOhtDbd7Rr1ghE9yiF6Oq Ig32grj40g3bdASlQGMqd9DdQ1RdCNjbX00wuQOHtO+zsNdMDhJI9fawK87epwjmKJQl+zS+ Wq6VFMew1367UAr4y3QApy/QNsDhapCkE8= IronPort-HdrOrdr: A9a23:6ialU6pqAN9Q8htfNvRwRocaV5vHL9V00zEX/kB9WHVpm5Oj+P xGzc526farslsssREb+OxpOMG7MBThHLpOkPMs1NaZLXLbUQ6TQr2KgrGSoQEIdxeOk9K1kJ 0QDpSWa+eAc2SS7/yKmTVQeuxIqLLskNHKuQ6d9QYUcegDUdAe0+4TMHf8LqQZfngjOXJvf6 Dsmvav6gDQMEg/X4CePD0oTuLDr9rEmNbPZgMHPQcu7E2rgSmz4LD3PhCE1lNGOgk/iosKwC zgqUjU96+ju/a0xlv10HLS1Y1fnJ/ExsFYDMKBp8AJInHHixquZq5mR7qe1QpF6N2H2RIPqp 3hsh0gN8N85zf4eXy0mwLk303a3DMn+xbZuCulqEqmhfa8aCMxCsJHi44cWADe8VAcsNZ117 8O936FtrJMZCmw0BjV1pztbVVHh0C0qX0tnao4lHpES7YTb7dXsMg24F5VKpEdByj3gbpXXd WGNPusqsq+TGnqLkww5gJUsZyRtzUIb127q3E5y4OoO2M8pgE786MarPZv60vouqhNCaWs3N 60QpiApIs+P/P+UpgNd9vpYfHHfVAlEii8Rl57HzzcZdM60jT22tvK3Ik= X-IronPort-AV: E=Sophos;i="5.90,151,1643691600"; d="scan'208";a="65382780" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Q6soH5wgoM6FT/v6XNIGsyX1yy7/mPj/2z/us1dT7n2tks2evagVNK8aVUxkyhn7GYTPAMOt1Dz9y+xy9PkmFaHeJjjuyQsOhlVa80+D95G/xJeIsJpkkfAPu4DgDnqZHCVLPdh5AOh2rwc4H4RDaG4xXM+oFNHQDsFFaDg/q3RMHlNfdzy43zSOtb8gU/sqUyTrvUcDFau2e/XMeMccUqkDyxLKXCrY0c6llR/3vrE7KfmL3b/pYueJ0W3LuF5eLp1sKCJwEgwMJlgwmSVU2ucqBhNa5rejPSUsFsZKsea1HwHMwldXcUVMMmGhkb8mujMDj7sjmgPyCLbq9XpgyQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=HxoMZSpShgbJE4AvL17XZo/c/OpnQlva5Wp9It6p7g8=; b=VtHLzrImjYqAt3FZxwnOYjyWbiuGdOahzlTOE9T541adUesyVqq0/KDCMmG5TwCHVRaRId1aLnyCe40kF7OJZGrEZIi3DND8+RKm9C6JAd0DObidA0LvBwipA9M4BvquXRPxuvTrDXqOEsuajnQCLe1lHvk0Fyk3eKTmja0igM+SkUfJkF0A5SNC/qMAla6j4UlDtR1B+kRM05qlAiDw5fsJgl4cy6OIu83UIa1s2R+U/YQH7EGoAih4LxLCLcbCYJ8I0ZFOEOHMOZP+zo8gHfw8BfbFuKiUSdw9NDq+ijNxk1zIqkk+nsOHVe9o9eHXNlXTdy0+tNiUcokuzJlJBw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com; dkim=pass header.d=citrix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=HxoMZSpShgbJE4AvL17XZo/c/OpnQlva5Wp9It6p7g8=; b=ohKKGqL6ynyFoiY4pcXA1Ps9QPerVeXFj8NNFXqTLiwjMWOctLi2nxS8DIMANgBg3aBcQrpvh8f8/sp01t8OheMmxUyIKfmnCugYwShI0/AOZxpBvLNVfK/gQpbCaDfKz77d53QREVbXWWDUGuqD+eiStSim5spxJIWPrrfV3bA= Date: Thu, 3 Mar 2022 11:39:43 +0100 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Ze Dupsys CC: , Subject: Re: ZFS + FreeBSD XEN dom0 panic Message-ID: References: <202203011540.221FeR4f028103@nfbcal.org> Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: LO2P265CA0252.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:8a::24) To DS7PR03MB5608.namprd03.prod.outlook.com (2603:10b6:5:2c9::18) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ce43f619-9df3-4c60-6656-08d9fd02237f X-MS-TrafficTypeDiagnostic: BY5PR03MB5156:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: WLisN88rwuxDaDFvLjzu3ZB2gOLtbYaCfID5KkOnc9PrH+TEE/fOxAJPx5TIMy/cXiUAjQQLjGDMAG96/X9qdYcAkbqFoKgZqsuvpe+X4Ld4EAbaBZxeJLg9FHmP1jCGk0CJJ2L45sw3Xzx7PkNtp3N6m8iScMHifrD+C3JB00bmD+wEKEugTQ4r8u7iYkZOeayWmTaBl27/XlzaYiBfGTS7LtLLIcVyG/Cy+U0kT4x+k/AOFiWMeU3XTGppQsl0GYOuDPz4r/tECO8CwhkTJ4EI8YZflX54LI+I9MGK0/YTfa/ODS52E+WpZKzST/oKgdWWOtpRxqLTSUrAo7QEGwjV+yUKbWNz/WQmthfnIfPO5KZ+UI+7Sfc13xMaCh8FTIc5DDIzFc5KKoljxSsGLL38ZCDK4g2JxVEpRL2dr19Wf3rNIUJhLOvJUZG5J3xoT9aKPtGhczjp60BPopr1oVO8DR7xKNBNOqGmJ/UhDTMsoCQr8iOSLaQfdtpgI71fY2WPJMMt7QKawBvKbMj+xVadqzfhqrxENuVTMpt46oS2k7v/+KK9Alwe5HqTCTtXTH2zWm0Ami/LWcv5L15QBou8sM1tilMmp7GnAEQb8U4njMaKF12cpZrO33uelj6kbGnDVNLAZZNrgcFxH/BBvmvMYWnw7tYFrjlpDbK3ymup913UoKBkZcQ+mSJj7OAAc4s3yiitCpWyqrTBqOtZsA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(7916004)(4636009)(366004)(85182001)(6666004)(6506007)(33716001)(6916009)(38100700002)(316002)(8936002)(2906002)(26005)(186003)(82960400001)(508600001)(66476007)(83380400001)(9686003)(4326008)(86362001)(6486002)(5660300002)(6512007)(66946007)(66556008)(8676002)(67856001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?MUE5Slk5U3hlaFhIYjNIMzlReFoxbjZ6NFF2K3lvL1htN2lDWVExK0d0UmVl?= =?utf-8?B?RmdjaWNQdjRGVU9FamtuellIYVNpckhvMElDT2kyWFdlZ0hyeHg4R1ZrN0hM?= =?utf-8?B?UW4zMjgvdHhxbWJ2Z1pGQUlwNUduOUpSUVlqRnpaZmJyOWFoMkNmK3BMUWEr?= =?utf-8?B?SjN0RTRuZ3J0RWcvQnVXTzdDOVVWY0JvaVBtWTNaK3NiWDhDbE1aek5EZE9W?= =?utf-8?B?Tis4MEVvQkdFWHVzQUJoZW5CWW8wZTgvMENHL1ppb09xTDlqUGZLejBTeHNo?= =?utf-8?B?VVl3bDNjbCtFMEtXbXJOT0VlamV1VmVqM1hSYlRSOFgwSDJTZXF1MzRxMDEr?= =?utf-8?B?ZWhJMDdUU0RqdHZ2NHd5N0hpZGlwS0h2eFVOQXo1U3E2SVBXVHRCb09hYmtC?= =?utf-8?B?bWNuczFMdGpHNDhyaWdialdXVEZtTzVoaUVZeFZNdGZkRlExdlA1RFd3djZl?= =?utf-8?B?WTFkZTlxR2ZiQ1VGT3BpSHVUam96YWxpa1ZUY2E0QjNDZlFPSGtxem5GNExt?= =?utf-8?B?cDZiK3M1a3FVNXdBelBhWS9SRVlUMmlvM1hlSFgzckZVaUV6VjQvN1RoQ2l0?= =?utf-8?B?OUtHb21TelhJSCtlWWdoOGw3MkhYNm03ckRucVkzSEhGallocTloMGVQNzZ4?= =?utf-8?B?ekRMMlJ2ZVFkSkxyNEVHTVl6Y2lHTVNaR2tBby8wdWlvdS8raTlwTkF4VmtQ?= =?utf-8?B?SjFYNzhjd0prTnA0VFN3bTAyRVltZ0dwb21oUkIzbjdJZUhIY2VEaG81dnNn?= =?utf-8?B?SkJQOW5rNmcwZ2FkSE8yU1UxQzNkaTc0YkpDZ0tzeE1tRkw2dVZTWWt5VmJi?= =?utf-8?B?eXNnWDBNUVo3SXZjTzdVb1dheDQ2OFhoaXpnZVR3czlOZXRhMVJBUjBZYmk5?= =?utf-8?B?VEUxajlWSS95Kzl1YURpRE12RGJraEtmRGVmVkxmNGNpdXUzelYyZlU2UGtT?= =?utf-8?B?Q2VXWXlVWklJVkQ2UlpmaC9NZEZ1N3JkeVlPWVlLaVIvTkhvQk1qQVI3ZVN3?= =?utf-8?B?SkprLzRQSzQ4RnF5Q3dUMk1QYXFoZGIzdjd1OUkxdWlsMTVpakhDaThaUmJv?= =?utf-8?B?MGowSzNUV3RYZjN6ZGdEc2dYOXJHM3pJT0Z0d1JYQWJRalJIRkx1cHZTdmQy?= =?utf-8?B?WTJya2FmWEVpN3o1K09hdEI0SUlLeWc1UU14NXdGbGJqVWhlb1laeWlLZFlX?= =?utf-8?B?ejF6aHRkVzlLUERzSmZJZE9FS1B5ZURMSWdtWVRYb05KVkJ5a2lTWmpVSm5N?= =?utf-8?B?NHpScWN6TmFDVUJzLzhyTXFlQVA3YTJTMDhpMDV3ZkJBWGZUNHQzVlpKOWhr?= =?utf-8?B?b1VPazRQZzJ5eTl2a3hmNFJhSlo0dkJwUVhtZnozVjNEaVJ5SG1JcElDWHd2?= =?utf-8?B?dEQzdFVjdGEwL2tKdy9EWVl2UmQweWxDR2dhSlE0Y3VONVFITXI2cWRvNlV5?= =?utf-8?B?SEZESW1sSnl0a2F0bDV3akgxL1hkbTdEZDlsWEdqTzkyV1M2cnVsdUNpMEFr?= =?utf-8?B?ZXFpVjMxS3hRSkg4VWxqa1NBRThxdnFmNFpkZ2RsajNmUENvakZqRDYrUkw0?= =?utf-8?B?OG01Tm5lRWxNTnBIM05vZ2VQUGp0MWVteGI2SjUwZ3RuajkwU3o3YlA3M1F5?= =?utf-8?B?eXVSeDc5T21TNVA2OG9WR0hCQkRGRTRzbUxEVklXVFNvZ3RzY0NOVVBhU2hu?= =?utf-8?B?dkhVMVZxaTVPRnFIOU94VXVQWjRvamhyLzhScjdwclR5by9HRTdzamprSFpm?= =?utf-8?B?K0YwcG5XeE1UUkptcm0yZThobWVONWx5b2xEOTdlZmIwWDJxblo5RzYxaENk?= =?utf-8?B?eFozZlJJNldZekYzUFB1bktZV0tvSUt0VjlZZ0phWG5IZzZnRk1SRm15RXdL?= =?utf-8?B?Wk9rTHFLbkJOcFRpS2I4VzZrdlcrKzZqeVFrOXlHeG50TUZNN25raXQrbTBw?= =?utf-8?B?R2xrMlVRTGhhNVZRSm1qYUJTSEw4TmFwSEcxUjJ2Q0p3c29ndVROWkhuN1JG?= =?utf-8?B?S29HbzRBNEwvcWd3enZ3UlhHT1hyZFhvL3VxMXNEVUpQVTV4SVozcStjTk16?= =?utf-8?B?TVZJd0dVQ050WkgwU1J1bEpLTDBpckhyQkZUS09aV1hiR1dwMHRaTnVqRW9h?= =?utf-8?B?Q0IxU1RPd3pvNTJZWlUxUUlibjlucU1tU3QrQ0EyQzUwQkU1U0t5T0xTSklj?= =?utf-8?Q?f7KBPMMdaXAeSZCnuEpKJ0A=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: ce43f619-9df3-4c60-6656-08d9fd02237f X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Mar 2022 10:39:47.9136 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: gFG1JGFzKjSKTtB0k73b/iOcph6e1fv3asAiU6NTujhFL8ULPz5RWSqsNijNS0x5yIx7+SaLtgmTq2/6eCf3IA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR03MB5156 X-OriginatorOrg: citrix.com X-Rspamd-Queue-Id: 4K8SDM6TyTz4jsB X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=citrix.com header.s=securemail header.b=NEAIt31B; dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=ohKKGqL6; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=reject) header.from=citrix.com; spf=pass (mx1.freebsd.org: domain of "prvs=05474af58=roger.pau@citrix.com" designates 216.71.145.153 as permitted sender) smtp.mailfrom="prvs=05474af58=roger.pau@citrix.com" X-Spamd-Result: default: False [-5.64 / 15.00]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+exists:216.71.145.153.spf.hc3370-68.iphmx.com]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; RCVD_IN_DNSWL_MED(-0.20)[216.71.145.153:from]; DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+]; DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_COUNT_ZERO(0.00)[0]; MIME_TRACE(0.00)[0:+]; R_MIXED_CHARSET(0.56)[subject]; ASN(0.00)[asn:16417, ipnet:216.71.145.0/24, country:US]; FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=05474af58=roger.pau@citrix.com]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; FORGED_SENDER_VERP_SRS(0.00)[]; DWL_DNSWL_LOW(-1.00)[citrix.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[freebsd-xen]; RWL_MAILSPIKE_POSSIBLE(0.00)[216.71.145.153:from]; MID_RHS_NOT_FQDN(0.50)[] X-ThisMailContainsUnwantedMimeParts: N On Wed, Mar 02, 2022 at 07:26:18PM +0200, Ze Dupsys wrote: > Today managed to crash lab Dom0 with: > xen_cmdline="dom0_mem=6144M dom0_max_vcpus=2 dom0=pvh,verbose=1 > console=vga,com1 com1=9600,8n1 guest_loglvl=all loglvl=all sync_console=1 > reboot=no" Hm, it's weird that reboot=no doesn't work for you. Does noreboot instead make a difference? > > I wrote ' vmstat -m | sort -k 2 -r' each 120 seconds, the latest one was as > in attachment, panic was with the same fingerprint as the one with > "rman_is_region_manager" line already reported. > The scripts i ran in parallel generally were the same as attached in bug > report, just a bit modified. > 1) ./libexec.sh zfs_volstress_fast_4g (this just creates new ZVOLs and > instead of 2GB, it writes 4BG in each ZVOL created dd if=/dev/zero) > 2) ./test_vm1_zvol_3gb.sh (this loops commands: start first DomU, write > 3GB in it's /tmp, restart DomU, removes /tmp, repeat) > 3) ./test_vm2_zvol_5_on_off.sh (this loops: start second DomU, which has 5 > disks attached, turn off DomU, repeat) Right. So the trigger for this seem to be related to creating (and destroying) VMs in a loop? Do you still see the same if you only execute steps 1 and 4 from the repro described above? > 4) monitoring, sleep 120 seconds, print vmstat | sort in serial output. > > Around dom id 108, system started to behave suspiciously, xl list showed > DomUs created, but they did not really start up, script timeout-ed for ssh > connection, no vnc. When i did xl destroy manually, and xl create, system > panic happened. Could you also add the output of `top -n1` to see where memory is going? I'm quite sure we have a leak in some of the backends, maybe the bounce buffer used by blkback. Thanks, Roger.