Re: About PHYS_TO_DMAP

Mitchell Horne <mhorne@freebsd.org> writes:

> On 4/16/23 04:06, Dmitry Salychev wrote:
>> Mitchell Horne <mhorne@freebsd.org> writes:
>> 
>>> On 4/14/23 04:31, Dmitry Salychev wrote:
>>>> Hi,
>>>> I'm struggling to understand which KVA will be returned by
>>>> PHYS_TO_DMAP
>>>> on arm64. For example, if I'll create a DMA tag this way:
>>>> 	bus_dma_tag_create(
>>>> 	    bus_get_dma_tag(dev),
>>>> 	    sc->buf_align, 0,		/* alignment, boundary */
>>>> 	    DMAP_MAX_PHYSADDR,		/* low restricted addr */
>>>> 	    DMAP_MIN_PHYSADDR,		/* high restricted addr */
>>>
>>> I think you are confused about the purpose of lowaddr and
>>> highaddr. They specify the window of bus-space addresses that are
>>> *inaccessible* *to the device* for DMA. It does not prevent you from
>>> providing a buffer backed by physical memory within this range, but in
>>> this case bounce pages will be used as an intermediate to perform the
>>> DMA transaction.
>>>
>>> Most commonly, if the device can only address a 32-bit range, then the
>>> values lowaddr=BUS_SPACE_MAXADDR_32BIT and highaddr=BUS_SPACE_MAXADDR
>>> will be given. If the entire bus range is accessible to the device for
>>> DMA, a zero-sized window will be given: lowaddr=BUS_SPACE_MAXADDR,
>>> highaddr=BUS_SPACE_MAXADDR.
>>>
>>> This is all described in adequate detail in busdma(9), but it is still
>>> not easily understood without a lot of code study, in my experience.
>>>
>> According to busdma(9), the window contains all addresses *greater
>> than*
>> lowaddr and *less than or equal to* highaddr, i.e. your example with the
>> 32-bit range looks like:
>>                allowed                      prohibited
>>      ------------------------------(xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>
>>                              lowaddr                           highaddr
>>              BUS_SPACE_MAXADDR_32BIT                  BUS_SPACE_MAXADDR
>> But my example looks a bit different:
>>         prohibited              allowed                 prohibited
>>      xxxxxxxxxxxxxxxx]----------------------------(xxxxxxxxxxxxxxxxxxx>
>>               highaddr                      lowaddr
>>      DMAP_MIN_PHYSADDR            DMAP_MAX_PHYSADDR
>> 
>
> Okay I see, you are using MAX as lowaddr and MIN as highaddr. I read
> it backwards the first time.
>
> However, I looked at the code and I'm not sure that it will handle the
> case where lowaddr > highaddr correctly. Just a warning to you.
>
>> I've found the only example of such DMA tag configuration in
>> src/sys/dev/vnic/nicvf_queues.c at line 379 (rbdr_buff_dmat).
>>
> I think this example is misleading, if not incorrect. All DMA
> transactions for this tag will have a destination (physical memory
> address) that falls within the [DMAP_MIN_PHYSADDR,DMAP_MAX_PHYSADDR]
> range. So, the specified boundaries are not meaningful, and don't
> represent a real constraint of the device. At least, this is my
> understanding of it.

I don't know the bus_dma implementation details yet unfortunately and
thought that DMA tag created with [DMAP_MIN_PHYSADDR,DMAP_MAX_PHYSADDR]
window is required to calculate KVA by PHYS_TO_DMAP properly.

However, PHYS_TO_DMAP gave me an address to access frame annotation even
with DMA tag created without a restriction window at all, i.e.
BUS_SPACE_MAXADDR used for both lowaddr and highaddr.

I've opened a review https://reviews.freebsd.org/D39946 if you'll be
interested in details.

>
>> 
>>>> 	    NULL, NULL,			/* filter, filterarg */
>>>> 	    BUF_SIZE, 1,		/* maxsize, nsegments */
>>>> 	    BUF_SIZE, 0,		/* maxsegsize, flags */
>>>> 	    NULL, NULL,			/* lockfunc, lockarg */
>>>> 	    &dmat);
>>>> in order to restrict any physical addresses but a window defined by
>>>> DMAP_MIN_PHYSADDR and DMAP_MAX_PHYSADDR. Later on when I'll be
>>>> mapping my mbuf (BUF_SIZE) with
>>>> 	bus_dmamap_load_mbuf_sg(dmat, dmap,
>>>> 	    m, &segs, &nsegs, BUS_DMA_NOWAIT);
>>>> I expect that
>>>> 	m->m_data == PHYS_TO_DMAP(segs[0].ds_addr)
>>>
>>> Why do you expect or need this to be the case?
>>>
>>> busdma is not responsible for setting or modifying m_data, it comes
>>> from wherever you allocated the mbuf. Calling
>>> bus_dmamap_load_mbuf_sg() will prepare a DMA mapping where the m_data
>>> buffer is used as the source/destination for the DMA transaction, but
>>> busdma does not allocate the buffer itself.
>>>
>> I don't need this to be the case exactly, but I'd like to be able to
>> access a "frame annotation" (64 bytes long) which is located exactly at
>> the start of m_data buffer having that physical address is provided, i.e.
>> 	fa = (struct dpaa2_fa *) mbuf->m_data;
>>          /* ... fa populated ... */
>>          /* ... DMA transaction of the Tx frame (together with fa) ... */
>>          /* ... Tx confirmation from HW (bus address of the frame
>> only) ...*/
>>          fa = (struct dpaa2_fa *) PHYS_TO_DMAP(paddr);
>> 
>
> The short answer is that you are using PHYS_TO_DMAP correctly. Every
> real paddr should have a corresponding VA in the DMAP that is valid
> for read/write access to that memory. However, it is unusual to see
> DMAP usage in driver code.
>
> It is hard for me to say exactly what you should be doing without more
> information.
>
> In general, the m_data pointer remains valid after the DMA
> transaction. If your updated frame annotation is located at the
> beginning of this buffer, then you want to use the same m_data pointer
> to access it.
>
> I guess the challenge is that the TX confirmation occurs in a separate
> context (interrupt handler?) where the mbuf reference is lost? I
> believe the usual practice is to maintain some kind of ring or queue
> of transmitted mbuf+dma_map_t pairs, which can be handled together in
> the TX interrupt handler (unload DMA map, free and/or reassign mbuf).

This is correct. However, HW (firmware in my case) provides a frame
descriptor which carries physical address of the Rx/Tx buffer only. This
is why I needed a way to access the buffer itself from the kernel
address space.

>
> Of course, you must use appropriate sync calls before accessing the
> buffer again on the host, probably: bus_dmamap_sync(tag, map,
> BUS_DMASYNC_POSTREAD | BUS_DMASYNC_POSTWRITE). I'm sure you know this.
>
>>>> but it isn't true. Could somebody explain what exactly is returned
>>>> by
>>>> PHYS_TO_DMAP in this case and whether it's possible to translate
>>>> physical address to KVA as fast as possible (O(1) ideally).
>>>>
>>>
>>> PHYS_TO_DMAP is always a linear calculation of: physaddr + DMAP_MIN_ADDRESS.
>>>
>>> I do not think PHYS_TO_DMAP is in use at all in this example, or
>>> anywhere within busdma really.
>>>
>> Is there any other way to obtain KVA of a buffer mapped for DMA
>> transaction by the physical address? I've been crawling source code for
>> sometime already, but DMAP is the only thing I managed to find.
>>  >>> Regards,
>>>> Dmitry
>>>> --
>>>> Open source software/hardware enthusiast
>>>> hackaday.io/dsl | github.com/mcusim | patreon.com/salychev
>>>>
>> Regards,
>> Dmitry

-- 
Open source software/hardware enthusiast
hackaday.io/dsl | github.com/mcusim | patreon.com/salychev