bhyve and vfs.zfs.arc_max, and zfs tuning for a hypervisor

[Matt Churchyard]

Patrick M. Hausen wrote:
> 
> > 1. Does ARC actually cache zfs volumes (not files/datasets)?
> 
> Yes it does.

I find this distinction between volumes/files/etc and what is cached causes confusion (as well as "volumes not datasets").

Both ZVOLs and Z file systems are types of dataset. A dataset stores data in records (usually up to 128kb in size).
It's these records that are cached (and that most ZFS functions such as compression/raidz/zil/etc work with)

As far as the ZFS lower levels are concerned, there is no difference between a volume and a file system.

> 
> > 2. If ARC does cache volumes, does this cache make sense on a 
> > hypervisor, because guest OSes will probably have their own disk cache anyway.
> 
> IMHO not much, because the guest OS is relying on the fact that when 
> it writes it’s own cached data out to „disk“, it will be committed to 
> stable storage.

Maybe I've missed something but I don't quite get the link between read cache (ARC) and guest writes here?

> This is an important point.

> > 3. Would it make sense to limit vfs.zfs.arc_max to 1/8 or even less 
> > of total RAM, so that most RAM is available to guest machines?
> 
> Yes if you build your own solution on plain FreeBSD. No if you are 
> running FreeNAS which already tries to autotune the ARC size according 
> to the memory committed to VMs.
> 
> > 4. What other zfs tuning measures can you suggest for a bhyve 
> > hypervisor?
> 
> e.g.
> 	zfs set sync=always zfs/vm
> 
> if zfs/vm is the dataset under which you create the ZVOLs for your 
> emulated disks.

>Well, bhyve already has an option for this:

>The block-device-options are:

>nocache   Open the file with O_DIRECT.
>direct    Open the file using O_SYNC.
>ro        Force the file to be opened read-only.

>I think something like
>"-s 4:0,virtio-blk,/dev/zvol/zroot/vm/mail/disk0,direct"
>would do the same?

> 
> I’m using this for all my VM „disks“ and have added a 16 GB SLOG 
> device to my spinning disk pool - seems to work great. This is on a home system.

> Is SLOG also used by zfs volumes?

As above, the core of ZFS doesn't care what type of dataset it is working with. ARC/ZIL/etc all work exactly the same.

--
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
2:5005/49 at fidonet http://vas.tomsk.ru/