From nobody Mon Apr 24 14:43:36 2023 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q4nty1gTyz46vkX for ; Mon, 24 Apr 2023 14:43:38 +0000 (UTC) (envelope-from mhorne@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q4ntx6TbLz3pvv for ; Mon, 24 Apr 2023 14:43:37 +0000 (UTC) (envelope-from mhorne@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1682347417; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/sFI2/117sGbbj1elBynR6pWTvelPcwywRduBG5LpJ4=; b=Oa15CRQFk2kPaFNuC6fx/DY8gWpuZwIRUuta/G/mW9abxF+wwSr/RkpaksI31jWGOro08y URWW8SrEONz53u18MnZr6kwdQUgfTgUcG5PFzSIdjSA4M0COXR6xcAymqUnM9M9FUwC44R Zxxsf+SFsnFwzcievOBtiC+5sx0dbfyjfckTQrgctUYRuiJVI6WDSDM3DbRMGJp/B3HBZu e9uHdiAakzeeJDo92BjDdH+jRtxq4KsX/OrIqA0nQtkEO9RQbGfIbIlaMiuN0iETHBGJJF caU5+7x/h70hdT8FgR9CuDIF+dK/663S45dYbO+EYnSqTSaxqNCIXDmFBYQQhA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1682347417; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/sFI2/117sGbbj1elBynR6pWTvelPcwywRduBG5LpJ4=; b=S4xKlGbjLNNNX6aIQwFdRrm8vXbWRaDqVgqJq+sLYKv4jf1OH3uf3GXzzdehHL39K51zip b/JqwY6QcwvDgZU4JJrrCrDTBC951f19ro4F9IkuTJpEH8GWYp7Xu4dQPYU6AAOaK/43BG XHhPB9MlZzYJSwv3iKHGv2+G0Bm+Bw/zRKuvyu0TpE8T1Ukez1H17FhW+ZnThATjCq2GAg fvuR3hOJimZoZqo8NNdKRE91gfgG00xlZvqWG0hsmZ0AUU/b4tFtB8hfeOHmY3k9qaUh6R okyJ6JMV4v9Fd2dfxd9eyAWT+CsmBkuHbdsVgS8L1qNIycoCAI5YAoBiwXLOjw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1682347417; a=rsa-sha256; cv=none; b=Ps+HpBXVArEuE3jQYg+/CVFaDVVPUHYYM0u2OuCb7KtiQHK5CgHfolJJkFlT76ezIxhwIE AbnuTke9gydRMVkJHh5PD0N7CE21P1+e71KRlv3l0f+mTQfi2VLFl+TNcoZRvId5wYx5qU x5ecQEtvun17MMd9Hnt/3/is2YAQUHuP7SMkEXTtqxg0mjHwxj4UXTqUW6gIe81dVZgDmE 7zaG4NXHbqhylc+jfCfl5N24VsqyjdVOqqW8KieX5/TG9RRohddaOM8qSNV0KtrUCEnQyD XGIEUBrGesSRs+ty2jblFDlkJ53Q5/G6+qr0pxBcux1TOx+SMG7WvRljqJCEZw== Received: from [192.168.1.151] (host-173-212-76-127.public.eastlink.ca [173.212.76.127]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: mhorne) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Q4ntx4hKZzfB7 for ; Mon, 24 Apr 2023 14:43:37 +0000 (UTC) (envelope-from mhorne@freebsd.org) Message-ID: <08ff62fd-89db-d429-9ef3-04f213118f83@freebsd.org> Date: Mon, 24 Apr 2023 11:43:36 -0300 List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.10.0 Subject: Re: About PHYS_TO_DMAP To: freebsd-arm@freebsd.org References: <86ildyucuv.fsf@peasant.tower.home> <86zg78s2bj.fsf@peasant.tower.home> Content-Language: en-CA From: Mitchell Horne In-Reply-To: <86zg78s2bj.fsf@peasant.tower.home> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ThisMailContainsUnwantedMimeParts: N On 4/16/23 04:06, Dmitry Salychev wrote: > > Mitchell Horne writes: > >> On 4/14/23 04:31, Dmitry Salychev wrote: >>> Hi, >>> I'm struggling to understand which KVA will be returned by >>> PHYS_TO_DMAP >>> on arm64. For example, if I'll create a DMA tag this way: >>> bus_dma_tag_create( >>> bus_get_dma_tag(dev), >>> sc->buf_align, 0, /* alignment, boundary */ >>> DMAP_MAX_PHYSADDR, /* low restricted addr */ >>> DMAP_MIN_PHYSADDR, /* high restricted addr */ >> >> I think you are confused about the purpose of lowaddr and >> highaddr. They specify the window of bus-space addresses that are >> *inaccessible* *to the device* for DMA. It does not prevent you from >> providing a buffer backed by physical memory within this range, but in >> this case bounce pages will be used as an intermediate to perform the >> DMA transaction. >> >> Most commonly, if the device can only address a 32-bit range, then the >> values lowaddr=BUS_SPACE_MAXADDR_32BIT and highaddr=BUS_SPACE_MAXADDR >> will be given. If the entire bus range is accessible to the device for >> DMA, a zero-sized window will be given: lowaddr=BUS_SPACE_MAXADDR, >> highaddr=BUS_SPACE_MAXADDR. >> >> This is all described in adequate detail in busdma(9), but it is still >> not easily understood without a lot of code study, in my experience. >> > > According to busdma(9), the window contains all addresses *greater than* > lowaddr and *less than or equal to* highaddr, i.e. your example with the > 32-bit range looks like: > > > allowed prohibited > ------------------------------(xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx> > lowaddr highaddr > BUS_SPACE_MAXADDR_32BIT BUS_SPACE_MAXADDR > > But my example looks a bit different: > > prohibited allowed prohibited > xxxxxxxxxxxxxxxx]----------------------------(xxxxxxxxxxxxxxxxxxx> > highaddr lowaddr > DMAP_MIN_PHYSADDR DMAP_MAX_PHYSADDR > Okay I see, you are using MAX as lowaddr and MIN as highaddr. I read it backwards the first time. However, I looked at the code and I'm not sure that it will handle the case where lowaddr > highaddr correctly. Just a warning to you. > I've found the only example of such DMA tag configuration in > src/sys/dev/vnic/nicvf_queues.c at line 379 (rbdr_buff_dmat). > I think this example is misleading, if not incorrect. All DMA transactions for this tag will have a destination (physical memory address) that falls within the [DMAP_MIN_PHYSADDR,DMAP_MAX_PHYSADDR] range. So, the specified boundaries are not meaningful, and don't represent a real constraint of the device. At least, this is my understanding of it. > >>> NULL, NULL, /* filter, filterarg */ >>> BUF_SIZE, 1, /* maxsize, nsegments */ >>> BUF_SIZE, 0, /* maxsegsize, flags */ >>> NULL, NULL, /* lockfunc, lockarg */ >>> &dmat); >>> in order to restrict any physical addresses but a window defined by >>> DMAP_MIN_PHYSADDR and DMAP_MAX_PHYSADDR. Later on when I'll be >>> mapping my mbuf (BUF_SIZE) with >>> bus_dmamap_load_mbuf_sg(dmat, dmap, >>> m, &segs, &nsegs, BUS_DMA_NOWAIT); >>> I expect that >>> m->m_data == PHYS_TO_DMAP(segs[0].ds_addr) >> >> Why do you expect or need this to be the case? >> >> busdma is not responsible for setting or modifying m_data, it comes >> from wherever you allocated the mbuf. Calling >> bus_dmamap_load_mbuf_sg() will prepare a DMA mapping where the m_data >> buffer is used as the source/destination for the DMA transaction, but >> busdma does not allocate the buffer itself. >> > > I don't need this to be the case exactly, but I'd like to be able to > access a "frame annotation" (64 bytes long) which is located exactly at > the start of m_data buffer having that physical address is provided, i.e. > > fa = (struct dpaa2_fa *) mbuf->m_data; > /* ... fa populated ... */ > /* ... DMA transaction of the Tx frame (together with fa) ... */ > > /* ... Tx confirmation from HW (bus address of the frame only) ...*/ > fa = (struct dpaa2_fa *) PHYS_TO_DMAP(paddr); > The short answer is that you are using PHYS_TO_DMAP correctly. Every real paddr should have a corresponding VA in the DMAP that is valid for read/write access to that memory. However, it is unusual to see DMAP usage in driver code. It is hard for me to say exactly what you should be doing without more information. In general, the m_data pointer remains valid after the DMA transaction. If your updated frame annotation is located at the beginning of this buffer, then you want to use the same m_data pointer to access it. I guess the challenge is that the TX confirmation occurs in a separate context (interrupt handler?) where the mbuf reference is lost? I believe the usual practice is to maintain some kind of ring or queue of transmitted mbuf+dma_map_t pairs, which can be handled together in the TX interrupt handler (unload DMA map, free and/or reassign mbuf). Of course, you must use appropriate sync calls before accessing the buffer again on the host, probably: bus_dmamap_sync(tag, map, BUS_DMASYNC_POSTREAD | BUS_DMASYNC_POSTWRITE). I'm sure you know this. >>> but it isn't true. Could somebody explain what exactly is returned >>> by >>> PHYS_TO_DMAP in this case and whether it's possible to translate >>> physical address to KVA as fast as possible (O(1) ideally). >>> >> >> PHYS_TO_DMAP is always a linear calculation of: physaddr + DMAP_MIN_ADDRESS. >> >> I do not think PHYS_TO_DMAP is in use at all in this example, or >> anywhere within busdma really. >> > > Is there any other way to obtain KVA of a buffer mapped for DMA > transaction by the physical address? I've been crawling source code for > sometime already, but DMAP is the only thing I managed to find. > >>> Regards, >>> Dmitry >>> -- >>> Open source software/hardware enthusiast >>> hackaday.io/dsl | github.com/mcusim | patreon.com/salychev >>> > > Regards, > Dmitry