From nobody Thu Mar 10 10:34:07 2022 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 460761A17EC5 for ; Thu, 10 Mar 2022 10:33:51 +0000 (UTC) (envelope-from zedupsys@gmail.com) Received: from mail-lf1-x134.google.com (mail-lf1-x134.google.com [IPv6:2a00:1450:4864:20::134]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KDlly1CTWz4ntq for ; Thu, 10 Mar 2022 10:33:50 +0000 (UTC) (envelope-from zedupsys@gmail.com) Received: by mail-lf1-x134.google.com with SMTP id 3so8623172lfr.7 for ; Thu, 10 Mar 2022 02:33:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=subject:to:cc:references:reply-to:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding:content-language; bh=9Ba3Y5MjouiJ1K44xxNH0WlDGC1RP/RK0P0KjfA2sdI=; b=K4xQUsip/jtxqXJkhRtNyxzAz+K4JS2MbgbbvmIKhcWUGhiFJQZUjMU15E46E3nvhu p7f9c3SsTqlOLuEIAJX8uvdtRuDSrlJCSiWN+RWZX9+K5tJM92EqON4fd1mNMqDWRYdg ML2maYOfIeFTy83y4Hsqdu/nV5vUwM8rs/klcLnNHRwKxw3EnyRdZrXSpPh01Iub69m4 UMeqxR2n9nlq2hawpxfs8HzZWDTPTY4HdtyyFQ8+/YhsuptDVhNEAMopK665K7mBH+Ul CTfYm0mydpJqkTBn+NUR+YyAuazYMbcQ2AZEDIj3oPjjAZv4pq79ne0iup3YsAlcGn2Q iONg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:subject:to:cc:references:reply-to:from :message-id:date:user-agent:mime-version:in-reply-to :content-transfer-encoding:content-language; bh=9Ba3Y5MjouiJ1K44xxNH0WlDGC1RP/RK0P0KjfA2sdI=; b=rXCzaXKUx9I5MXtUAxIrgUzX2zMusfzNlqx+ginUiRPd3sikICIPHCAnxhtvViU/FR NVj3is6LrkBv3IPstAZhlu8+Gw36FJzVKt4vGV9UPwnpwUtG8jm+lvcF1WNjz9e2+ThH cMazNeBdrSHtM4M/Mx5rNaOfWxKT9PHFP1ovE0Q5wisSP/osAO5WZI6dHOVSKQYswOqb fbgkj9v3pMWO5Ue7qC6VrdRXa0IcE/zeserby0LcCpxzp2IeefcViXhRqy/76gcKuVr/ G2sgM361ASCkflu+welC006WEkgjC6dcNQZdePNq/WJuJrQ+p4b+tGU+BMc3PtugLMwL ZXaA== X-Gm-Message-State: AOAM531xtI1BoeUtIlaf4liy4XqQsNu5LeySDAahJNDI0+bkIfJluCUw qWyHw+uEARhKy72tHkUbcN6gAnv1cGY= X-Google-Smtp-Source: ABdhPJytLbiDc00ZdUCRjOeDOFzZHe2UpnLfTY2AnRynrb2qnSjRBdmHU2HPWWX0VTXcrWiX3+b9Sg== X-Received: by 2002:ac2:5de4:0:b0:448:2daf:b98c with SMTP id z4-20020ac25de4000000b004482dafb98cmr2477838lfq.641.1646908428691; Thu, 10 Mar 2022 02:33:48 -0800 (PST) Received: from [10.3.0.1] ([213.110.65.3]) by smtp.googlemail.com with ESMTPSA id z4-20020a0565120c0400b00443823361d0sm901146lfu.243.2022.03.10.02.33.47 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 10 Mar 2022 02:33:48 -0800 (PST) Subject: Re: ZFS + FreeBSD XEN dom0 panic To: Brian Buhrow Cc: freebsd-xen@freebsd.org References: <202203022007.222K7Y7I013387@nfbcal.org> Reply-To: zedupsys@gmail.com From: Ze Dupsys Message-ID: <1937e88f-159a-5324-db8f-687e84a0bea9@gmail.com> Date: Thu, 10 Mar 2022 12:34:07 +0200 User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.12.0 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 In-Reply-To: <202203022007.222K7Y7I013387@nfbcal.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Rspamd-Queue-Id: 4KDlly1CTWz4ntq X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=K4xQUsip; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of zedupsys@gmail.com designates 2a00:1450:4864:20::134 as permitted sender) smtp.mailfrom=zedupsys@gmail.com X-Spamd-Result: default: False [-3.29 / 15.00]; HAS_REPLYTO(0.00)[zedupsys@gmail.com]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; REPLYTO_ADDR_EQ_FROM(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.29)[-0.289]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; FREEMAIL_REPLYTO(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::134:from]; MLMMJ_DEST(0.00)[freebsd-xen]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N On 2022.03.02. 22:07, Brian Buhrow wrote: > Hello. Given Roger's message and the presentation of the errors you're seeing, I think > we're both in agreement that you're running out of memory. Below I've listed some suggestions > on how you might go about making your system more stable, and, possibly figuring out where the > trouble is, exactly. When I was building my infrastructure, I ran into a bunch of edge > conditions and weird bugs as well, so some ideas that may not seem obvious, may, in fact, work, > simply because they cause you to skirt some latent bug in the system. > > Here's what my setup looks like, in case it helps. > > > Hope these notes are helpful. Hello Brian, Yes, of course those notes are helpful. Knowing that somebody has a working and stable setup and how his configuration/setup is, is a help. Thank you for the detailed information. I did not respond right away, because i wanted to follow those steps before writing anything and i did not have a spare HDD for "new" testing system. Wanted to still keep already set-up current version 13.0 for testing as well. For testing i made somewhat simplified setup of your described one: 1. Installed FreeBSD-12.1, UFS ada0p1, brand new HDD, basic configuration just for network, resolve.conf, etc. 2. 8GB RAM Dom0, 16GB swap on ada0p2 3. i did not use gmirror for now, since it is a test setup and on this new HDD problems should not happen so soon, 4. bridge0 created, no vlans, just a simple bridge, no firewall not even physical NIC bridged, just for VMs 5. creating zpool on ada1, mountpoint=none, here though i got a question: When i create ZFS partition on disk, i use "gpart add -t freebsd-zfs .." and similar for UFS, but what gpt type should be used for XEN DomU passthrough partition? I mean, if i passed through whole disk to DomU, and VM formated it as ZFS, in case of physical system reboot, BIOS could get confused and maybe even boot from that previously passed-through-HDD. AFAIK FreeBSD boot loader prefers ZFS over UFS if both available. If gpt type is freebsd-zfs, how can i ensure that Dom0 does not try to taste it or mount it? In a way i know that in VM the given block device will be formatted with MBR or GPT, but still, what gpt type should be used in that case, so that Dom0 is satisfied and it is somewhat secure. For ZFS volumes for now i use volmode=dev. 6. well, to crash sooner i created 1 VM with 1 HDD, and other with 5 HDDs, passsed trhough ZVOLs. After approx. 10 hours of running given scripts, on/off 2 VMs and load on ZFS, machine panic'ed, so my hopes for stable setup were shattered. The only difference with your written mail was that on FreeBSD 12.1 RELEASE pkg repo, there is only xen-kernel-4.12 available. I will try to run with VMs with 1 and 2 HDDs only, but historically i know that FreeBSD 12.2 with 6GB RAM crashed with those setups as well, it just took longer to happen. Another difference is hardware used. Do you remember if you have tuned some BIOS settings? I'm thinking to poke around there as well. Could you share a config template for one of DomU VMs? Maybe i use some bad syntax for HDD devices or something. Thank you!