PIM_UNMAPPED question

[Max Gurtovoy]

Hi,

I am working on a new iscsi RDMA transport implementation. In order to 
avoid kernel buffer allocation in the IO path
I added support for unmapped buffers in my driver (PIM_UNMAPPED hba_misc 
flag).
This indeed helped performance by improving the IOPs By a factor of 4x 
vs. KVAs.

However, I noticed that the bio_ma vector consists of physically 
contiguous addresses. I expected that user-space
allocations would not be physically contiguous. Is it possible that I'm 
still not seeing the user IO buffers?

I tested using dd/fio traffic generators and the phenomenon still persists.

128k Read output to show the phenomenon:

kernel: dma_address 0x40608000 len 4096
kernel: dma_address 0x40609000 len 4096
kernel: dma_address 0x4060a000 len 4096
kernel: dma_address 0x4060b000 len 4096
kernel: dma_address 0x4060c000 len 4096
kernel: dma_address 0x4060d000 len 4096
kernel: dma_address 0x4060e000 len 4096
kernel: dma_address 0x4060f000 len 4096
kernel: dma_address 0x40610000 len 4096
kernel: dma_address 0x40611000 len 4096
kernel: dma_address 0x40612000 len 4096
kernel: dma_address 0x40613000 len 4096
kernel: dma_address 0x40614000 len 4096
kernel: dma_address 0x40615000 len 4096
kernel: dma_address 0x40616000 len 4096
kernel: dma_address 0x40617000 len 4096
kernel: dma_address 0x40618000 len 4096
kernel: dma_address 0x40619000 len 4096
kernel: dma_address 0x4061a000 len 4096
kernel: dma_address 0x4061b000 len 4096
kernel: dma_address 0x4061c000 len 4096
kernel: dma_address 0x4061d000 len 4096
kernel: dma_address 0x4061e000 len 4096
kernel: dma_address 0x4061f000 len 4096
kernel: dma_address 0x40620000 len 4096
kernel: dma_address 0x40621000 len 4096
kernel: dma_address 0x40622000 len 4096
kernel: dma_address 0x40623000 len 4096
kernel: dma_address 0x40624000 len 4096
kernel: dma_address 0x40625000 len 4096
kernel: dma_address 0x40626000 len 4096
kernel: dma_address 0x40627000 len 4096

Initiator machine info:
- OS 11-current r284921
- 32G RAM
- 32 Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz CPUs

The code is available at:
https://github.com/sagigrimberg/iser-freebsd/tree/iser-rebase-11-current-r284921

Thanks in advanced,

Max Gurtovoy.