bhyve: corrupting zfs pools?

[Andriy Gapon]

I am doing a simple experiment.

I get FreeBSD image from here:
ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/ISO-IMAGES/11.0/FreeBSD-11.0-CURRENT-amd64-r283577-20150526-memstick.img.xz

Then I run in bhyve with two additional "disks" created with truncate -s 4g:
$ bhyveload -m 1G -d
~/tmp/FreeBSD-11.0-CURRENT-amd64-r283577-20150526-memstick.img test
$ bhyve -A -HP -s 0:0,hostbridge -s 1,lpc -s 2:0,virtio-net,tap0 -s
3:0,virtio-blk,/home/avg/tmp/FreeBSD-11.0-CURRENT-amd64-r283577-20150526-memstick.img
-s 3:1,virtio-blk,/tmp/l2arc-test/hdd1,sectorsize=512/4096 -s
3:2,virtio-blk,/tmp/l2arc-test/hdd2,sectorsize=512/4096 -l com1,stdio -l
com2,/dev/nmdm0A -c 2 -m 1g test

Note sectorsize=512/4096 options.  Not sure if it's them that cause the trouble.

Then, in the VM:
$ zpool create l2arc-test mirror /dev/vtbd1 /dev/vtbd2
$ zfs create -p l2arc-test/ROOT/initial
$ tar -c --one-file-system -f - / | tar -x -C /l2arc-test/ROOT/initial -f -

Afterwards, zpool status -v reports no problem.
But then I run zpool scrub and get the following in the end:
$ zpool status -v
  pool: l2arc-test
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: http://illumos.org/msg/ZFS-8000-8A
  scan: scrub repaired 356K in 0h0m with 9 errors on Tue Jun  2 13:58:17 2015
config:

        NAME        STATE     READ WRITE CKSUM
        l2arc-test  ONLINE       0     0     9
          mirror-0  ONLINE       0     0    18
            vtbd1   ONLINE       0     0    25
            vtbd2   ONLINE       0     0    23

errors: Permanent errors have been detected in the following files:

        /l2arc-test/ROOT/initial/usr/bin/svnlitesync
        /l2arc-test/ROOT/initial/usr/freebsd-dist/kernel.txz
        /l2arc-test/ROOT/initial/usr/freebsd-dist/src.txz

/l2arc-test/ROOT/initial/usr/lib/clang/3.6.1/lib/freebsd/libclang_rt.asan-x86_64.a


The same issue is reproducible with ahci-hd.

My host system is a recent amd64 CURRENT as well.  The hardware platform is AMD.

-- 
Andriy Gapon