Why is there no bus_dmamap_load_sg()?

[Pratyush Yadav]

Hi Warner,

On Sat 26 May, 2018, 7:40 PM Warner Losh, <imp at bsdimp.com> wrote:

>
>
> On Sat, May 26, 2018 at 3:07 AM, Pratyush Yadav <pratyush at freebsd.org>
> wrote:
>
>> I was looking at the man page and I notice the function
>> bus_dmamap_load_mbuf_sg() which directly returns the segment array to
>> the client. But a similar variant of bus_dmamap_load() is missing.
>> This variant would be useful to me and help me in my work.
>>
>> Is there a reason the sg variant was only implemented for mbufs?
>>
>> I was looking at the code in sys/kern/subr_bus_dma.c and a
>> bus_dmamap_load_sg() does not look too difficult to implement. Is it
>> ok if I submit a patch implementing it?
>>
>
> The biggest reason is because of the failure modes. If there's not
> sufficient memory or other resources to honor the request, we defer until
> there is such memory / resources and then call the callback. There's no
> convenient hook the drivers can connect to so they can retry the allocation
> when memory becomes available (busdma has one since eventually a request
> will complete and unload). The original port was for CAM SIMs which had
> stringent requirements to always make progress on I/Os to avoid deadlock.
> Rather than put this tricky code in all the CAM SIMs, it's centralized to
> retry as soon and as robustly as it can.
>
> Network drivers, however, don't have such requirements and often don't
> care. If they can't load a map, they can put the packet back on the
> transmit queue (or whatever), set a flag and return without putting it into
> the NIC's buffer rings. Eventually, a packet will finish transmitting and
> there will be a chance to send it again (or one of a number of other
> events, I'm glossing heavily here). It's a better fit for network drivers
> than storage drivers. Since bus_dmamap_load_mbuf* is used only for network
> drivers, it makes sense.
>
> So, it would only make sense to add if you can cope with BUS_DMA_NOWAIT
> always, and that error handling is simpler than letting busdma call you
> back when the shortage is over. Is that the case for your port?
>

I'm not sure. I looked at the OpenBSD code and they always call with a
BUS_DMA_NOWAIT flag, but I'm not sure what the answer is for FreeBSD.

Or is it just because OpenBSD doesn't have this sort of interface (the
> callback difference dates back to the initial port to FreeBSD by Justin
> Gibbs because it never got folded back into NetBSD for reasons too long,
> too old and only half remembered by me for this post)? Can the code you are
> porting cope with error returns and retry again later?
>

The other problem with using callbacks is that because I pass the
xen-specific callback to the dma load, the driver using the xen specific
dma operation can't pass it's own callback. I thought using the _sg()
variant would allow the driver to use a callback. But the way you put it,
that kind of defeats the purpose of the callback, which is to use it when
allocation is defered.

The bottom line is, I need to re-evaluate. I will spend some more time on
this and discuss it with my mentors. I'll let you (and everyone else) know
if I do decide we need it.

Thanks for the enlightening answer.

Regards,
Pratyush Yadav