bhyve: zvols for guest disk - yes or no?

[Jan Bramkamp]

On 16/11/2016 15:47, Patrick M. Hausen wrote:
> Hi, all,
>
> we are just starting a project that will run a couple
> of Ubuntu guests on top of bhyve instead of ESXi
> that we used in the past.
>
> As far as I could find out, more or less all bhyve
> manager/wrapper tools use zvols as the backing
> store for "raw" guest disk images.
>
> I looked at
>
> 	* chyves
> 	* iohyve
> 	* vm-bhyve
>
> So far so good. Yet, this blog article has some very
> valid (IMHO) points against using them:
>
> http://jrs-s.net/2016/06/16/psa-snapshots-are-better-than-zvols

Afaik this is only a problem if you create a reservation for your ZVOLs. 
By default ZFS does create ZVOLs with reservations matching their size. 
The problem is that there is no way to signal to a VM that the 
overcommited SAN just ran out of space and your stupid operator is 
looking for ways to increase free space *NOW* and the VM should just 
retry. Instead the VM gets a general write error. To keep this promise 
ZFS has to reserve enough space to change each bit in the ZVOL if you 
take a snapshot of a ZVOL. Other datasets are created without 
reservations by default because the POSIX file system API can report 
ENOSPC if the file system ran out of space. Of course you get the same 
problem if you storage VM images in a file system there is just no sane 
default setting protecting you, because you just outsmarted your poor 
operating system.

> Another thing I'm pondering is: wouldn't it be better to
> run on UFS so you can dedicate as much memory
> as possible to VMs?

ZFS isn't a memory eating boogie man. Yes ZFS uses a lot of memory and 
requires more memory than UFS for stable performance, but you get a lot 
in return:

  * End to end checksumming
  * Snapshots
  * Remote replication of snaphosts
  * Painless volume management
  * ...

> So I'm a bit puzzled now on what to do. Any opinions,
> experiences, war stories to share?

ZFS saved my bacon more than once. Twice it detected and corrected 
datacorruption on dying hardware in time. With a normal FS both cases 
would have turned into bitrot spreading into the backups until it's too 
late.

Without ZFS you would require a reliable hardware RAID controller (if 
such a magical creature exists) instead (or build a software RAID1+0 
from gmirror and gstripe). IMO money is better invested into more RAM 
keeping ZFS and the admin happy.

-- Jan Bramkamp