Re: About PHYS_TO_DMAP

On 4/16/23 04:06, Dmitry Salychev wrote:
> 
> Mitchell Horne <mhorne@freebsd.org> writes:
> 
>> On 4/14/23 04:31, Dmitry Salychev wrote:
>>> Hi,
>>> I'm struggling to understand which KVA will be returned by
>>> PHYS_TO_DMAP
>>> on arm64. For example, if I'll create a DMA tag this way:
>>> 	bus_dma_tag_create(
>>> 	    bus_get_dma_tag(dev),
>>> 	    sc->buf_align, 0,		/* alignment, boundary */
>>> 	    DMAP_MAX_PHYSADDR,		/* low restricted addr */
>>> 	    DMAP_MIN_PHYSADDR,		/* high restricted addr */
>>
>> I think you are confused about the purpose of lowaddr and
>> highaddr. They specify the window of bus-space addresses that are
>> *inaccessible* *to the device* for DMA. It does not prevent you from
>> providing a buffer backed by physical memory within this range, but in
>> this case bounce pages will be used as an intermediate to perform the
>> DMA transaction.
>>
>> Most commonly, if the device can only address a 32-bit range, then the
>> values lowaddr=BUS_SPACE_MAXADDR_32BIT and highaddr=BUS_SPACE_MAXADDR
>> will be given. If the entire bus range is accessible to the device for
>> DMA, a zero-sized window will be given: lowaddr=BUS_SPACE_MAXADDR,
>> highaddr=BUS_SPACE_MAXADDR.
>>
>> This is all described in adequate detail in busdma(9), but it is still
>> not easily understood without a lot of code study, in my experience.
>>
> 
> According to busdma(9), the window contains all addresses *greater than*
> lowaddr and *less than or equal to* highaddr, i.e. your example with the
> 32-bit range looks like:
> 
> 
>                allowed                      prohibited
>      ------------------------------(xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>
>                              lowaddr                           highaddr
>              BUS_SPACE_MAXADDR_32BIT                  BUS_SPACE_MAXADDR
> 
> But my example looks a bit different:
> 
>         prohibited              allowed                 prohibited
>      xxxxxxxxxxxxxxxx]----------------------------(xxxxxxxxxxxxxxxxxxx>
>               highaddr                      lowaddr
>      DMAP_MIN_PHYSADDR            DMAP_MAX_PHYSADDR
> 

Okay I see, you are using MAX as lowaddr and MIN as highaddr. I read it 
backwards the first time.

However, I looked at the code and I'm not sure that it will handle the 
case where lowaddr > highaddr correctly. Just a warning to you.

> I've found the only example of such DMA tag configuration in
> src/sys/dev/vnic/nicvf_queues.c at line 379 (rbdr_buff_dmat).
>
I think this example is misleading, if not incorrect. All DMA 
transactions for this tag will have a destination (physical memory 
address) that falls within the [DMAP_MIN_PHYSADDR,DMAP_MAX_PHYSADDR] 
range. So, the specified boundaries are not meaningful, and don't 
represent a real constraint of the device. At least, this is my 
understanding of it.

> 
>>> 	    NULL, NULL,			/* filter, filterarg */
>>> 	    BUF_SIZE, 1,		/* maxsize, nsegments */
>>> 	    BUF_SIZE, 0,		/* maxsegsize, flags */
>>> 	    NULL, NULL,			/* lockfunc, lockarg */
>>> 	    &dmat);
>>> in order to restrict any physical addresses but a window defined by
>>> DMAP_MIN_PHYSADDR and DMAP_MAX_PHYSADDR. Later on when I'll be
>>> mapping my mbuf (BUF_SIZE) with
>>> 	bus_dmamap_load_mbuf_sg(dmat, dmap,
>>> 	    m, &segs, &nsegs, BUS_DMA_NOWAIT);
>>> I expect that
>>> 	m->m_data == PHYS_TO_DMAP(segs[0].ds_addr)
>>
>> Why do you expect or need this to be the case?
>>
>> busdma is not responsible for setting or modifying m_data, it comes
>> from wherever you allocated the mbuf. Calling
>> bus_dmamap_load_mbuf_sg() will prepare a DMA mapping where the m_data
>> buffer is used as the source/destination for the DMA transaction, but
>> busdma does not allocate the buffer itself.
>>
> 
> I don't need this to be the case exactly, but I'd like to be able to
> access a "frame annotation" (64 bytes long) which is located exactly at
> the start of m_data buffer having that physical address is provided, i.e.
> 
> 	fa = (struct dpaa2_fa *) mbuf->m_data;
>          /* ... fa populated ... */
>          /* ... DMA transaction of the Tx frame (together with fa) ... */
> 
>          /* ... Tx confirmation from HW (bus address of the frame only) ...*/
>          fa = (struct dpaa2_fa *) PHYS_TO_DMAP(paddr);
> 

The short answer is that you are using PHYS_TO_DMAP correctly. Every 
real paddr should have a corresponding VA in the DMAP that is valid for 
read/write access to that memory. However, it is unusual to see DMAP 
usage in driver code.

It is hard for me to say exactly what you should be doing without more 
information.

In general, the m_data pointer remains valid after the DMA transaction. 
If your updated frame annotation is located at the beginning of this 
buffer, then you want to use the same m_data pointer to access it.

I guess the challenge is that the TX confirmation occurs in a separate 
context (interrupt handler?) where the mbuf reference is lost? I believe 
the usual practice is to maintain some kind of ring or queue of 
transmitted mbuf+dma_map_t pairs, which can be handled together in the 
TX interrupt handler (unload DMA map, free and/or reassign mbuf).

Of course, you must use appropriate sync calls before accessing the 
buffer again on the host, probably: bus_dmamap_sync(tag, map, 
BUS_DMASYNC_POSTREAD | BUS_DMASYNC_POSTWRITE). I'm sure you know this.

>>> but it isn't true. Could somebody explain what exactly is returned
>>> by
>>> PHYS_TO_DMAP in this case and whether it's possible to translate
>>> physical address to KVA as fast as possible (O(1) ideally).
>>>
>>
>> PHYS_TO_DMAP is always a linear calculation of: physaddr + DMAP_MIN_ADDRESS.
>>
>> I do not think PHYS_TO_DMAP is in use at all in this example, or
>> anywhere within busdma really.
>>
> 
> Is there any other way to obtain KVA of a buffer mapped for DMA
> transaction by the physical address? I've been crawling source code for
> sometime already, but DMAP is the only thing I managed to find.
>  >>> Regards,
>>> Dmitry
>>> --
>>> Open source software/hardware enthusiast
>>> hackaday.io/dsl | github.com/mcusim | patreon.com/salychev
>>>
> 
> Regards,
> Dmitry