How to send 1MB I/O size in a single I/O request without split

[Sibananda Sahu]

Hey Scoot,

Thanks very much for your reply.

>> -          How can I get more sge count in an I/O request?
>>

>If you want to test multiple segments, I suggest leaving you system running
>for a long time with multiple processes freeing and allocating memory so
>that the system becomes >fragmented.  You can also modify the bus_dma_tag
>in your driver to specify a maximum segment size of 4k instead of
>(presumably in your case) something larger.  That will force >busdma to
>stop merging adjacent segments.

As you have said:

"The busdma API will see that the allocation is contiguous and attempt to
merge the contiguous segments.  This is usually desirable since few segments
reduces processing overhead in the driver and the hardware."

If I will modify the bus_dma_tag in our driver to have a max segment of 4K
then what is the performance impact?


Thanks,
Sibananda Sahu


-----Original Message-----
From: Scott Long [mailto:scott4long at yahoo.com]
Sent: Friday, January 30, 2015 1:47 AM
To: Sibananda Sahu
Cc: freebsd-scsi at freebsd.org
Subject: Re: How to send 1MB I/O size in a single I/O request without split


> On Jan 29, 2015, at 11:56 AM, Sibananda Sahu
> <sibananda.sahu at avagotech.com> wrote:
>
> Hi All,
>
>
>
> Recently we have added large I/O size of 1MB in our LSI controller and
> for that we have implemented the same in driver.
>
> But I have observed that the large I/O that an application is able to
> send is 128KB in one I/O request.
>
>
>
> I used the following command to send 1MB I/O:
>
> # ddpt if=/dev/da0 of=/dev/zero count=1 bs=1M
>
>
>
> But I have observed that the number of scatter gather elements per I/O
> request always comes 1 and the I/O length comes as 128KB(max).
>
> […]
>
>
> So my primary questions are:
>
> -          How can I send a large I/O size of 1MB in a single I/O request
> without any split?

You answered this question already, you must redefine MAXPHYS.  This can be
done via a kernel compile option, and many users and companies already know
how to do this.  I plan to start a discussion on increasing the default
size.

>
> -          Why I am getting always 1 scatter gather element?
>

You are likely getting an allocation that is physically contiguous.  This is
especially true since you are using ‘dd’ from userland, and the memory
allocator in userland tries to useon 2MB superpages for allocations.  The
busdma API will see that the allocation is contiguous and attempt to merge
the contiguous segments.  This is usually desirable since few segments
reduces processing overhead in the driver and the hardware.

> -          How can I get more sge count in an I/O request?
>

If you want to test multiple segments, I suggest leaving you system running
for a long time with multiple processes freeing and allocating memory so
that the system becomes fragmented.  You can also modify the bus_dma_tag in
your driver to specify a maximum segment size of 4k instead of (presumably
in your case) something larger.  That will force busdma to stop merging
adjacent segments.

Scott