Re: ZFS + FreeBSD XEN dom0 panic

On 2022.03.02. 22:07, Brian Buhrow wrote:
> 	Hello.  Given Roger's message and the presentation of the errors you're seeing, I think
> we're both in agreement that you're running out of memory.  Below I've listed some suggestions
> on how you might go about making your system more stable, and, possibly figuring out where the
> trouble is, exactly.  When I was building my infrastructure, I ran into a bunch of edge
> conditions and weird bugs as well, so some ideas that may not seem obvious, may, in fact, work,
> simply because they cause you to skirt some latent bug in the system.
>
> Here's what my setup looks like, in case it helps.
>
>
> Hope these notes are helpful.

Hello Brian,

Yes, of course those notes are helpful. Knowing that somebody has a 
working and stable setup and how his configuration/setup is, is a help.

Thank you for the detailed information. I did not respond right away, 
because i wanted to follow those steps before writing anything and i did 
not have a spare HDD for "new" testing system. Wanted to still keep 
already set-up current version 13.0 for testing as well.

For testing i made somewhat simplified setup of your described one:
1. Installed FreeBSD-12.1, UFS ada0p1, brand new HDD, basic 
configuration just for network, resolve.conf, etc.
2. 8GB RAM Dom0, 16GB swap on ada0p2
3. i did not use gmirror for now, since it is a test setup and on this 
new HDD problems should not happen so soon,
4. bridge0 created, no vlans, just a simple bridge, no firewall not even 
physical NIC bridged, just for VMs
5. creating zpool on ada1, mountpoint=none, here though i got a question:
When i create ZFS partition on disk, i use "gpart add -t freebsd-zfs .." 
and similar for UFS, but what gpt type should be used for XEN DomU 
passthrough partition? I mean, if i passed through whole disk to DomU, 
and VM formated it as ZFS, in case of physical system reboot, BIOS could 
get confused and maybe even boot from that previously 
passed-through-HDD. AFAIK FreeBSD boot loader prefers ZFS over UFS if 
both available. If gpt type is freebsd-zfs, how can i ensure that Dom0 
does not try to taste it or mount it?

In a way i know that in VM the given block device will be formatted with 
MBR or GPT, but still, what gpt type should be used in that case, so 
that Dom0 is satisfied and it is somewhat secure. For ZFS volumes for 
now i use volmode=dev.

6. well, to crash sooner i created 1 VM with 1 HDD, and other with 5 
HDDs, passsed trhough ZVOLs.

After approx. 10 hours of running given scripts, on/off 2 VMs and load 
on ZFS, machine panic'ed, so my hopes for stable setup were shattered. 
The only difference with your written mail was that on FreeBSD 12.1 
RELEASE pkg repo, there is only xen-kernel-4.12 available. I will try to 
run with VMs with 1 and 2 HDDs only, but historically i know that 
FreeBSD 12.2 with 6GB RAM crashed with those setups as well, it just 
took longer to happen. Another difference is hardware used. Do you 
remember if you have tuned some BIOS settings? I'm thinking to poke 
around there as well.

Could you share a config template for one of DomU VMs? Maybe i use some 
bad syntax for HDD devices or something.

Thank you!