Re: when does BIO_FLUSH happen on a disk?

On Fri, Jan 24, 2025 at 08:10:01PM +0100, Jonathan Krebs wrote:
> Dear FreeBSD developers,
> 
> Inside a QEMU-VM, I inserted calls to g_print_bio in the first lines of
> sys/cam/ata/ata_da.c:adastrategy() and
> sys/dev/virtio/block/virtio_blk.c:vtblk_strategy() of a FreeBSD 14.2, and
> added an image file with -drive file=test.img,format=raw[,if=virtio].
> 
> While using the virtual disk, I observe BIO_WRITE request, but never a
> BIO_FLUSH.
> Usage was by
>  - creating a UFS filesystem (newfs /dev/vtbd0)
>  - mounting it in /mnt
>  - write a file with
> 	dd of=/mnt/a bs=4k count=20 if=/dev/urandom
> 	sync
> 	dd of=/mnt/a bs=4k count=20 if=/dev/urandom  conv=fdatasync
> 	dd of=/mnt/a bs=4k count=20 if=/dev/urandom  oflag=sync
The only case where UFS issues BIO_FLUSH, together with BIO_ORDERED flag,
to prevent reordering of the requests before/after the flush, is when
a journal segment is written.  This is done to ensure that metadata blocks
are written after the conduit journal records are on the stable storage.

Generally, UFS uses softupdates technique and relies on the callbacks from 
the buffer write completions to update internal structures to indicate
metadata write progress.  With such approach, there is no need to issue
flushes.   For instance, when doing fsync(2) and scheduling writes required
to flush the metadata and data, we only return from syscall when disk
reported that all writes were done.

The case above (for the journal segment write) could be considered as a
way to short-cut some additional code that would otherwise need to be
written.

> 
>  - repeating this with a geli layer (no special parameters to geli init) and
> with msdosfs instead of UFS
> 
> The differences between the dd calls with and without sync options was that
> the WRITEs were immediately issued with conv=fdatasync or oflag=sync, but no
> FLUSHs were observed (with one exception: geli clear)
> 
> This is why I wonder when BIO_FLUSH is used and if it means something
> different.
> 
> I am interested for these reasons:
> 
> - In a university research project, I am trying to suspend+resume a system
> without calling the DEVICE_SUSPEND functions. Therefore I wonder when I can
> assume that a BIO_WRITE has really arrived on disk
> 
> - The GELI class seems to reorder WRITEs and FLUSHes. (at least in my
> understanding while looking for code example on writing a geom class) This
> was also discussed here 12 years ago, where I could not find a resolution: https://freebsd-geom.freebsd.narkive.com/KruBpS7r/geli-and-bio-flush-and-or-bio-ordered-issue
> (writes are queued, flush is passed g_down immediately,
> https://cgit.freebsd.org/src/tree/sys/geom/eli/g_eli.c#n507 )
> 
> Thanks and Happy Hacking,
> thejonny