splitting iovecs to bios

[Konstantin Belousov]

On Thu, Dec 10, 2015 at 02:20:19PM +0200, Max Gurtovoy wrote:
> Hi all,
> 
> I'm developing an iSER (iSCSI extensions for RDMA) driver for FreeBSD 
> 11-Current and I encounter some weired behaviour while testing IOs with 
> iovcnt > 1.
> I wrote a small test program that creates an iovec struct (let's say in 
> size of 4) and each iov_base starts from page begining and 4k len:
> 0 iov_base 0xe25000 iov_len 4096
> 1 iov_base 0xe26000 iov_len 4096
> 2 iov_base 0xe27000 iov_len 4096
> 3 iov_base 0xe28000 iov_len 4096
> 
> I use readv/writev to send this iovec data.
> I saw that my driver get 4 different bio requests even thought this 
> vector is aligned. This is surprising becasue in other OS I get only 1 bio.
> I have noticed the the physio function in sys/kern/kern_physio.c is 
> praparing one bio for each iovec entry.
> 
> is there a reason for this kind of implementation ? is there an option 
> to send this array using 1 bio (some flag) ?
> We can improve performance if we send it in 1 bio instead of 4.
> My driver supports BIO_UNMAPPED.

There might be indeed a reason, it could be that some drivers expect
blocking to be done by the userspace.  The drivers could have some
restrictions on transfer sizes and atomicity of transfer, which would
be broken by the unconditional merge.  I cannot give you an example
of such driver, known block-aware drivers like sa(4) only require the
bio size to be multiple of the basic block size.

OTOH, I see no issue with adding a SI_PHYSIOMERGE flag and doing the
merges for the driver in physio(), when unmapped request has consequtive
iov elements ending and starting at the page boundary.