Several bhyve quirks

[Peter Grehan]

Hi Julian,

  I'll let Neel take care of the time questions.

> 3. Even moderate guest disk I/O completely kills guest network
> performance.  For example, whenever security(8) (security(7) in FreeBSD)
> runs, guest network throughput drops from 150+ Mbps to ~20 Mbps, and
> jitter from ping jumps from <0.01 ms to 100+ ms.  If I try to build
> something in the guest, then network becomes almost unusable.
>
> The network performance degradation only affects the guest that's
> generating the I/O; high I/O on guest B doesn't affect guest A, nor
> would high I/O on the host.
>
> I'm using both virtio-blk and virio-net drivers, and the guests' disk
> images are backed by zvol+geli.  Removing geli has no effect.
>
> There are some commits in CURRENT that suggests improved virtio
> performance, but I'm not comfortable running CURRENT.  Is there a
> workaround I could use for 10.1?

  In 10.1, virtio-blk i/o is done sychronously in the context of the 
guest vCPU exit. If it's a single vCPU guest, or the virtio-net 
interrupt happens to be delivered to that vCPU, performance will suffer.

  A workaround is to use ahci-hd for the disk emulation and not 
virtio-blk. The AHCI emulation does i/o in a dedicated thread and 
doesn't block the vCPU thread.

> 4. virtio-blk always reports the virtual disk as having 512-byte
> sectors, and so I get I/O errors on OpenBSD guests when the disk image
> is backed by zvol+geli with 4K sector size.  Curiously, this only seems
> to affect zvol+geli; with just zvol it seems to work.  Also, it works
> either way on Linux guests.
>
> ATM I changed the zvol / geli sector size to 512 bytes, which probably
> made #2 worse.  I think this bug / feature is addressed by:
> <https://github.com/freebsd/freebsd/commit/02e846756ee99b849987a9bb6f57566fc70360c7>,
> but again is there a workaround to force a specific sector size for 10.1?

  The only workaround for 10.1 would be to use ahci-hd instead of 
virtio-blk. The correct sector size will be reported there.

> 5. This may be better directed at OpenBSD but I'll ask here anyway: if I
> enable virtio-rnd then OpenBSD would not boot with "couldn't map
> interrupt" error.  The kernel in bsd.rd will boot, but not the installed
> kernel (or the one built from STABLE; I forgot).  Again, Linux seems
> unaffected, but I couldn't tell if it's actually working.

  Try using the -W option to bhyve. This will force the bhyve virtio 
code to advertize (non-standard) MSI interrupt capability which OpenBSD 
will then use to allocate vectors.

later,

Peter.