From nobody Wed Mar 02 08:57:37 2022 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 0C39319FFD1E for ; Wed, 2 Mar 2022 08:57:07 +0000 (UTC) (envelope-from zedupsys@gmail.com) Received: from mail-pl1-x629.google.com (mail-pl1-x629.google.com [IPv6:2607:f8b0:4864:20::629]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4K7p022vyhz4SDl for ; Wed, 2 Mar 2022 08:57:06 +0000 (UTC) (envelope-from zedupsys@gmail.com) Received: by mail-pl1-x629.google.com with SMTP id e13so1012367plh.3 for ; Wed, 02 Mar 2022 00:57:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=QWxv9pzutJU4f9DsRNXmnx5ubZZIqANCeuUyp3xpNvY=; b=etOxfO8jrULJXB0JYnukC+dvrFvHJpOewkYdIG1pGGWrxxfliaNsbDAC8QLaN8r3y7 V+lTz7DQOhO+fC6jrgpwWxiLpD56QqfFQCWCqMVJekAbfLNYlVhuFdQ+EStT9iaWoJ38 fksZQhRLfwOjSfl3NQ8NKVxSt3J2lMR9B6G0HOXULr/CYTrOMJ2LQJyj+c3k75WXAJkT OoTyuQEtnp/eRX/GTRQPz6Lu3cjCKfDVM6BX4Jfvpogxf4Na531GA5wg5ZRUEruOYhQY Uvy11JmqYPU6oogCbgLcOp5q/8O0Yjpnb17LMLdGxFReacLkcd+WViHJa28glDQAqQBn FVYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=QWxv9pzutJU4f9DsRNXmnx5ubZZIqANCeuUyp3xpNvY=; b=JGxLXqQ51M8PDYk6VPCDXwjDxu2xQPL5E/iBHsLig8ExZwuZ52QGSk5D9JzV16L1rj Evt/cwoBknStS3e6GgKMXWliR9OjdyU6ns2JFfls3sD5LWdXXjFQZpK4DdEzCooyAnzj hrjYDgj3oTg/AJuWbDQTnzWc4b4V0BWV2/7sv3RP560O0YTD6mLEsJVcXP1OmV/dEYVt yxjMl0+5OjJvrcOHwi9TvJCEthsOa9PXvFK/OxO1njR/Ewl8Fdv/rPK2LWOLL4UDxQrA qOrsoPwqqCO4v2hkeBBTcEI/V6fjKlYWNMSr8gQA5IKysfkSqrBThQxgDE3jsUMkOmFv mUBA== X-Gm-Message-State: AOAM53137PavAV6v/0dVtHgF8xL1dPIK9MGPrSI2cj4YFlAG9UlTB74H sCxlrFEJDMWW6y4taJD4VcRiQwu13pJ9aiPALUB4PkFRCUg= X-Google-Smtp-Source: ABdhPJyLnulvOMW1i8mmqiYTpGBmqtuR9tU5XyJis+0HQsIN57MutE9cS25QMubtcI2CTj3mi3332h+qD4NDVgXIR0s= X-Received: by 2002:a17:902:ec86:b0:151:9ca2:8ecb with SMTP id x6-20020a170902ec8600b001519ca28ecbmr374535plg.61.1646211425383; Wed, 02 Mar 2022 00:57:05 -0800 (PST) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 References: <202203011540.221FeR4f028103@nfbcal.org> In-Reply-To: <202203011540.221FeR4f028103@nfbcal.org> From: Ze Dupsys Date: Wed, 2 Mar 2022 10:57:37 +0200 Message-ID: Subject: Re: ZFS + FreeBSD XEN dom0 panic To: freebsd-xen@freebsd.org, buhrow@nfbcal.org Content-Type: multipart/alternative; boundary="0000000000003c3a7005d93877a8" X-Rspamd-Queue-Id: 4K7p022vyhz4SDl X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=etOxfO8j; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of zedupsys@gmail.com designates 2607:f8b0:4864:20::629 as permitted sender) smtp.mailfrom=zedupsys@gmail.com X-Spamd-Result: default: False [-3.98 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.98)[-0.976]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org]; TO_DN_NONE(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::629:from]; MLMMJ_DEST(0.00)[freebsd-xen]; NEURAL_HAM_SHORT(-1.00)[-0.999]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-ThisMailContainsUnwantedMimeParts: N --0000000000003c3a7005d93877a8 Content-Type: text/plain; charset="UTF-8" Hello, I started using XEN on one pre-production machine (with aim to use later in production) with 12.2, but since it experienced random crashes i updated to 13.0 in hope that errors might disappear. I do not know how detailed should i write, so that this email is not too long, but gives enough info. FreeBSD Dom0 is installed on ZFS, somewhat basic install, IPFW and rules for NATting are used. Zpool is composed of 2 mirrored disks. There is a ZVOL volmode=dev for each VM and VM's jail that are attached as raw devices to DomU. At the moment DomUs contain FreeBSD, some 12.0 to 13.0, UFS, with VNET jails, epairs all bridged to DomU's xn0 interface. On Dom0 i have bridge interfaces, where DomU's are connected depending on their "zone/network", those that have allowed outgoing connections are NATted by IPFW on specific physical NIC and IP. xen_cmdline="dom0_mem=6144M cpufreq=dom0-kernel dom0_max_vcpus=4 dom0=pvh console=vga,com1 com1=115200,8n1 guest_loglvl=all loglvl=all" Physical hardware is XEON CPU, ECC RAM 16G, 2x8TB HDD. DomU config, something like this: memory = 1024 vcpus=2 name = "sys-01" type = "hvm" boot = "dc" vif = [ 'vifname=xbr0p5,type=vif,mac=00:16:3E:01:63:05,bridge=xbr0' ] disk = [ 'backendtype=phy, format=raw, vdev=xvda, target=/dev/zvol/sys/vmdk/root/sys-01-root', 'backendtype=phy, format=raw, vdev=xvdb, target=/dev/zvol/sys/vmdk/root/sys-01-jail1', 'backendtype=phy, format=raw, vdev=xvdc, target=/dev/zvol/sys/vmdk/root/sys-01-jail2' .. more defs, if any .. ] vnc=1 vnclisten="0.0.0.0:X" usbdevice = "tablet" serial = "pty" When just started, overall system works, speeds are acceptable, load is not high so system is not under stress. The thing is that at some unexpected times i noticed that system reboots, i.e. when i create new ZFS volume in Dom0, or when i reboot DomU or do something in Dom0 which seems unrelated, sometimes it was that init 0 would reboot system, sometimes it shut it down. It somehow felt, that panics happen when there is HDD load. So i got somewhat similar machine for testing/lab env, 16G ECC, slower XEON, 2x2TB HDD and serial port and started to try to push that system to limits with various combinations, restricting RAM, CPUs, etc. The bug info contains combination, that seemed for me to be the fastest way of how to panic system. For XEN startup "vifname=" did not work as described in XEN user manual pages for default startup script, so i added "ifconfig name $vifname" in that script. The necessity for it was, that ipfw rules that required "via $ifname in", had to have specific NIC, but XEN by default each time was creating new NIC name depending on which name was free. This is not active on lab system, and it still crashes, so i do not think that problem cause is this. About history. I believe hardware is okay, since before XEN i was using FreeBSD 12.2 (upgraded incrementally from 12.0), ZFS + jails a lot, VNETs used were netgraph(VNET bridge and ethernet interfaces). What i loved about that setup was, clean output of ifconfig, since host had only bridge interface and virtual ethernet interfaces for jails came directly from that bridge. New jail creation was just "zfs clone", it did not take much space, snapshots for backups could be made, whole HDD space could be easily expanded/limited for each jail, due to ZFS capabilities. System was stable. The problem with that setup was, that if some jail started to misbehave badly it was hard to control overall system performance and behavioral characteristics, i tried rctl, but jails could misbehave in new unexpected bad ways (exhausting RAM, process count, CPU load, HDD load, opening too many network sockets, etc. If OOM killer started to kill processes, it was impossible to control which process/jail should get killed first, which should be kept), so for me it seemed that virtualization is better way to solve that. I.e. to have a system VM, that has DNS, Web gateway, etc., and lower priority VMs, that could crash if misbehaving. I like XEN architecture in general, and i would like to use FreeBSD as Dom0, if possible; due to ZFS, knowledge and good history of OS stability. Since ZFS dataset can not be passed through to DomU, my idea was to use ZVOLs and UFS within VM, then i could snapshot those ZVOLs for backups, DomU could growfs when necessary. Somewhat less convenient as for jail architecture, but still, good enough. My first attempt was to keep netgraph jails in Dom0, but it turned out bad. Almost every time system panic happened when jail was started/stopped. Not first jail, but 5th+, panic-ed system with high probability. So i started to use epairs instead. It was less unstable, but still crashed from time to time. Now there are no jails, and still. I tried different ideas, to pass through whole HDD as raw in DomU-iscsi and use ctld on Dom0 to provide disks for other DomUs, HDD speed was bad, but system still crashed, i tried raw files on ZFS datasets, speeds seemed close to ZVOLs actually, but system still crashed. So now i was starting to wonder, what configurations do people use successfully? What have i missed? On Tue, Mar 1, 2022 at 5:40 PM Brian Buhrow wrote: > hello. I've been running FreeBSD-12.1 and Freebsd-12.2 plus ZFS > plus Xen with FreeBSD as > dom0 without any stability issues for about 2 years now. I'm doing this > on a number of > systems, with a variety of NetBSD, FreeBSD and Linux as domU guests. I > haven't looked at your > bug details, but are you running FreeBSD-13? > -thanks > -Brian > > --0000000000003c3a7005d93877a8 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello,

I started using XEN o= n one pre-production machine (with aim to use later in production) with 12.= 2, but since it experienced random crashes i updated to 13.0 in hope that e= rrors might disappear.

I do not know how detailed = should i write, so that this email is not too long, but gives enough info.<= br>

FreeBSD Dom0 is installed on ZFS, somewhat bas= ic install, IPFW and rules for NATting are used. Zpool is composed of 2 mir= rored disks. There is a ZVOL volmode=3Ddev for each VM and VM's jail th= at are attached as raw devices to DomU. At the moment DomUs contain FreeBSD= , some 12.0 to 13.0, UFS, with VNET jails, epairs all bridged to DomU's= xn0 interface. On Dom0 i have bridge interfaces, where DomU's are conn= ected depending on their "zone/network", those that have allowed = outgoing connections are NATted by IPFW on specific physical NIC and IP.

xen_cmdline=3D"dom0_mem=3D6144M cpufreq=3Ddom0-= kernel dom0_max_vcpus=3D4 dom0=3Dpvh console=3Dvga,com1 com1=3D115200,8n1 g= uest_loglvl=3Dall loglvl=3Dall"

Physical hard= ware is XEON CPU, ECC RAM 16G, 2x8TB HDD.

DomU con= fig, something like this:
memory =3D 1024
vcpus=3D2
name = =3D "sys-01"

type =3D "hvm"
boot =3D "dc= "

vif =3D [ 'vifname=3Dxbr0p5,type=3Dvif,mac=3D00:16:3E:01:= 63:05,bridge=3Dxbr0' ]
disk =3D [ 'backendtype=3Dphy, format=3Dr= aw, vdev=3Dxvda, target=3D/dev/zvol/sys/vmdk/root/sys-01-root',
=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0'backendtype=3Dphy, format=3Draw, vdev= =3Dxvdb, target=3D/dev/zvol/sys/vmdk/root/sys-01-jail1',
=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0'backendtype=3Dphy, format=3Draw, vdev=3Dxvdc, = target=3D/dev/zvol/sys/vmdk/root/sys-01-jail2'
=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 .. more defs, if any ..
= =C2=A0 =C2=A0 =C2=A0=C2=A0 ]

vnc=3D1
vnclisten=3D"0.0.0.0:X&= quot;
usbdevice =3D "tablet"
serial =3D "pty"
=


When just started, overall system = works, speeds are acceptable, load is not high so system is not under stres= s. The thing is that at some unexpected times i noticed that system reboots= , i.e. when i create new ZFS volume in Dom0, or when i reboot DomU or do so= mething in Dom0 which seems unrelated, sometimes it was that init 0 would r= eboot system, sometimes it shut it down. It somehow felt, that panics happe= n when there is HDD load. So i got somewhat similar machine for testing/lab= env, 16G ECC, slower XEON, 2x2TB HDD and serial port and started to try to= push that system to limits with various combinations, restricting RAM, CPU= s, etc. The bug info contains combination, that seemed for me to be the fas= test way of how to panic system.

For XEN start= up "vifname=3D" did not work as described in XEN user manual page= s for default startup script, so i added "ifconfig name $vifname"= in that script. The necessity for it was, that ipfw rules that required &q= uot;via $ifname in", had to have specific NIC, but XEN by default each= time was creating new NIC name depending on which name was free. This is n= ot active on lab system, and it still crashes, so i do not think that probl= em cause is this.


About history= .
I believe hardware is okay, since before XEN i was using FreeBS= D 12.2 (upgraded incrementally from 12.0), ZFS=C2=A0+ jails a lot, VNETs us= ed were netgraph(VNET bridge and ethernet interfaces). What i loved about t= hat setup was, clean output of ifconfig, since host had only bridge interfa= ce and virtual ethernet interfaces for jails came directly from that bridge= . New jail creation was just "zfs clone", it did not take much sp= ace, snapshots for backups could be made, whole HDD space could be easily e= xpanded/limited for each jail, due to ZFS capabilities. System was stable. = The problem with that setup was, that if some jail started to misbehave bad= ly it was hard to control overall system performance and behavioral charact= eristics, i tried rctl, but jails could misbehave in new unexpected bad way= s (exhausting RAM, process count, CPU load, HDD load, opening too many netw= ork sockets, etc. If OOM killer started to kill processes, it was impossibl= e to control which process/jail should get killed first, which should be ke= pt), so for me it seemed that virtualization is better way to solve that. I= .e. to have a system VM, that has DNS, Web gateway, etc., and lower priorit= y VMs, that could crash if misbehaving. I like XEN architecture in general,= and i would like to use FreeBSD as Dom0, if possible; due to ZFS, knowledg= e and good history of OS stability.

Since ZFS data= set can not be passed through to DomU, my idea was to use ZVOLs and UFS wit= hin VM, then i could snapshot those ZVOLs for backups, DomU could growfs wh= en necessary. Somewhat less convenient as for jail architecture, but still,= good enough.

My first attempt was to keep net= graph jails in Dom0, but it turned out bad. Almost every time system panic = happened when jail was started/stopped. Not first jail, but 5th+, panic-ed = system with high probability. So i started to use epairs instead. It was le= ss unstable, but still crashed from time to time. Now there are no jails, a= nd still.

I tried different ideas, to pass through= whole HDD as raw in DomU-iscsi and use ctld on Dom0 to provide disks for o= ther DomUs, HDD speed was bad, but system still crashed, i tried raw files = on ZFS datasets, speeds seemed close to ZVOLs actually, but system still cr= ashed. So now i was starting to wonder, what configurations do people use s= uccessfully? What have i missed?


On Tue, Mar 1, 202= 2 at 5:40 PM Brian Buhrow <buhrow@n= fbcal.org> wrote:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 hello.=C2=A0 I've been running F= reeBSD-12.1 and Freebsd-12.2 plus ZFS plus Xen with FreeBSD as
dom0=C2=A0 without any stability issues for about 2 years now.=C2=A0 I'= m doing this on a number of
systems, with a variety of=C2=A0 NetBSD, FreeBSD and Linux as domU guests.= =C2=A0 I haven't looked at your
bug details, but are you running FreeBSD-13?
-thanks
-Brian

--0000000000003c3a7005d93877a8--