ZFS corruption using zvols as backingstore for hvm VM's

Tue Nov 10 05:16:55 UTC 2020

	hello.   I'm running, or attempting to run, FreeBSD-12.1 with xen-4.12
with HVM machines, using Zvols as backingstore for the virtual disks.  I'm
running into an issue where after I write to the disk with the VM, 
filesystems on the disk become corrupted, including corruption of the disk
label and partition table.  zpool status shows no errors, even  after I
perform a scrub.   ZFS might be a coincidence and it may be a problem with
Xen or Qemu, which is why I'm writing here.  To do my install, I configured
two disks for my VM, one to use as a boot/root disk and the other to build
the system I'm virtualizing.  Interestingly enough, I do not see corruption
on the virtual boot disk, only on the secondary disk.  My config is shown
below.
	The VM is a NetBSD-5.2 system.  I'm running many 5.2 systems, both as
real hardware and as virtual machines without trouble, so there is
something about this setup that's the problem, as opposed to the OS in the
vm.  The dmesg for the VM is shown below as well.  Are secondary disks as
rw disks not supported?

-thanks
-Brian

Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
    2006, 2007, 2008, 2009, 2010
    The NetBSD Foundation, Inc.  All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
    The Regents of the University of California.  All rights reserved.

NetBSD 5.2_STABLE (ALTQ) #0: Fri Feb  8 01:00:12 PST 2019
	buhrow at lothlorien.nfbcal.org:/usr/src/sys/arch/i386/compile/ALTQ
total memory = 3839 MB
avail memory = 3761 MB
timecounter: Timecounters tick every 1.000 msec
timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
Xen HVM domU (4.12.1)
PCI BIOS rev. 2.1 found at 0xfd1c8
pcibios: config mechanism [1][x], special cycles [x][x], last bus 0
PCI IRQ Routing Table rev. 1.0 found at 0xf5d20, size 128 bytes (6 entries)
PCI Interrupt Router at 000:01:0 (Intel 82371FB (PIIX) PCI-ISA Bridge compatible)
mainbus0 (root)
cpu0 at mainbus0 apid 0: Intel 686-class, 3093MHz, id 0x206a7
cpu1 at mainbus0 apid 2: Intel 686-class, 3093MHz, id 0x206a7
ioapic0 at mainbus0 apid 1: pa 0xfec00000, version 11, 48 pins
acpi0 at mainbus0: Intel ACPICA 20080321
acpi0: X/RSDT: OemId <   Xen,     HVM,00000000>, AslId <HVML,00000000>
acpi0: SCI interrupting at int 9
acpi0: fixed-feature power button present
acpi0: fixed-feature sleep button present
timecounter: Timecounter "ACPI-Safe" frequency 3579545 Hz quality 900
ACPI-Safe 32-bit timer
hpet0 at acpi0 (HPET, PNP0103-0): mem 0xfed00000-0xfed003ff
timecounter: Timecounter "hpet0" frequency 62500000 Hz quality 2000
attimer1 at acpi0 (TMR, PNP0100): io 0x40-0x43 irq 0
pcppi1 at acpi0 (SPKR, PNP0800): io 0x61
midi0 at pcppi1: PC speaker (CPU-intensive output)
spkr0 at pcppi1
sysbeep0 at pcppi1
pckbc1 at acpi0 (PS2M, PNP0F13) (aux port): irq 12
pckbc2 at acpi0 (PS2K, PNP0303) (kbd port): io 0x60,0x64 irq 1
FDC0 (PNP0700) [PC standard floppy disk controller] at acpi0 not configured
UAR1 (PNP0501) [16550A-compatible COM port] at acpi0 not configured
apm0 at acpi0: Power Management spec V1.2
attimer1: attached to pcppi1
pckbd0 at pckbc2 (kbd slot)
pckbc2: using irq 1 for kbd slot
wskbd0 at pckbd0 mux 1
pms0 at pckbc2 (aux slot)
pckbc2: using irq 12 for aux slot
wsmouse0 at pms0 mux 0
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pchb0 at pci0 dev 0 function 0
pchb0: Intel 82441FX (PMC) PCI and Memory Controller (rev. 0x02)
pcib0 at pci0 dev 1 function 0
pcib0: Intel 82371SB (PIIX3) PCI-ISA Bridge (rev. 0x00)
piixide0 at pci0 dev 1 function 1
piixide0: Intel 82371SB IDE Interface (PIIX3) (rev. 0x00)
piixide0: bus-master DMA support present
piixide0: primary channel wired to compatibility mode
piixide0: primary channel interrupting at ioapic0 pin 14
atabus0 at piixide0 channel 0
piixide0: secondary channel wired to compatibility mode
piixide0: secondary channel interrupting at ioapic0 pin 15
atabus1 at piixide0 channel 1
piixpm0 at pci0 dev 1 function 3
piixpm0: Intel 82371AB (PIIX4) Power Management Controller (rev. 0x03)
timecounter: Timecounter "piixpm0" frequency 3579545 Hz quality 1000
piixpm0: 24-bit timer
piixpm0: SMBus disabled
XenSource, Inc. Xen Platform Device (undefined subclass 0x80, revision 0x01) at pci0 dev 2 function 0 not configured
vga1 at pci0 dev 3 function 0: Cirrus Logic CL-GD5446 (rev. 0x00)
wsdisplay0 at vga1 kbdmux 1
wsmux1: connecting to wsdisplay0
wskbd0: connecting to wsdisplay0
drm at vga1 not configured
wm0 at pci0 dev 4 function 0: Intel i82540EM 1000BASE-T Ethernet, rev. 3
wm0: interrupting at ioapic0 pin 32
wm0: 32-bit 33MHz PCI bus
wm0: 64 word (6 address bits) MicroWire EEPROM
wm0: Ethernet address 00:4e:46:42:42:ce
makphy0 at wm0 phy 1: Marvell 88E1011 Gigabit PHY, rev. 0
makphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
isa0 at pcib0
com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
com0: console
isapnp0 at isa0 port 0x279: ISA Plug 'n Play device support
npx0 at isa0 port 0xf0-0xff
npx0: reported by CPUID; using exception 16
fdc0 at isa0 port 0x3f0-0x3f7 irq 6 drq 2
isapnp0: no ISA Plug 'n Play devices found
timecounter: Timecounter "clockinterrupt" frequency 1000 Hz quality 0
timecounter: Timecounter "TSC" frequency 3093100120 Hz quality 3000
wd0 at atabus0 drive 0: <QEMU HARDDISK>
wd0: drive supports 16-sector PIO transfers, LBA48 addressing
wd0: 1045 GB, 2174281 cyl, 16 head, 63 sec, 512 bytes/sect x 2191675392 sectors
wd0: 32-bit data port
wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100)
wd1 at atabus0 drive 1: <QEMU HARDDISK>
wd1: drive supports 16-sector PIO transfers, LBA48 addressing
wd1: 1045 GB, 2174281 cyl, 16 head, 63 sec, 512 bytes/sect x 2191675392 sectors
wd1: 32-bit data port
wd1: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100)
wd0(piixide0:0:0): using PIO mode 4, DMA mode 2 (using DMA)
wd1(piixide0:0:1): using PIO mode 4, DMA mode 2 (using DMA)
IPsec: Initialized Security Association Processing.
Kernelized RAIDframe activated
boot device: wd0
root on wd0a dumps on wd0b
piixide0:0:0: lost interrupt
	type: ata tc_bcount: 512 tc_skip: 0
root file system type: ffs
wsdisplay0: screen 1 added (80x25, vt100 emulation)
wsdisplay0: screen 2 added (80x25, vt100 emulation)
wsdisplay0: screen 3 added (80x25, vt100 emulation)
wsdisplay0: screen 4 added (80x25, vt100 emulation)

Configuration file here

type = "hvm"

# Initial memory allocation (in megabytes) for the new domain.
memory = 4096

# Number of Virtual CPUS to use, default is 1
vcpus = 2

# A name for your domain. All domains must have different names.
name = "test_nbsd"

#----------------------------------------------------------------------------
# network configuration.
# The mac address is optional, it will use a random one if not specified.
# By default we create a bridged configuration; when a vif is created
# the script /usr/pkg/etc/xen/scripts/vif-bridge is called to connect
# the bridge to the designated bridge (the bridge should already be up)
vif = [  'type=ioemu,model=e1000,mac=00:4e:46:42:42:ce,bridge=bridge0' ]

#----------------------------------------------------------------------------
# Define the disk devices you want the domain to have access to, and
# what you want them accessible as.
# Each disk entry is of the form phy:UNAME,DEV,MODE
# where UNAME is the device, DEV is the device name the domain will see,
# and MODE is r for read-only, w for read-write.
# For NetBSD guest DEV doesn't matter, so we can just use increasing numbers
# here. For linux guests you have to use a linux device name (e.g. hda1)
# or the corresponding device number (e.g 0x301 for hda1)

disk = [ 
'/dev/zvol/xendisks/nb_backup,raw,hda,rw', 
'/dev/zvol/xendisks/nb_root,raw,hdb,rw' 
 ]

#Boot from the hard drive
boot = 'c'

# Turn off graphics
#sdl = 1
#Turn on the serial port as console
serial = 'pty'

#----------------------------------------------------------------------------
# Boot parameters (e.g. -s, -a, ...)
extra = ""

#============================================================================

#Reboot after shutdowns
#autorestart = True
on_poweroff = "restart"