From nobody Sun Apr 16 07:06:58 2023 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pzhsh21qKz44ptH for ; Sun, 16 Apr 2023 07:39:52 +0000 (UTC) (envelope-from dsl@mcusim.org) Received: from mcusim.org (mcusim.org [IPv6:2a00:dd80:3c::e63]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pzhsf5SZpz4QhV; Sun, 16 Apr 2023 07:39:50 +0000 (UTC) (envelope-from dsl@mcusim.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=mcusim.org header.s=default header.b=T3Js0gOU; spf=pass (mx1.freebsd.org: domain of dsl@mcusim.org designates 2a00:dd80:3c::e63 as permitted sender) smtp.mailfrom=dsl@mcusim.org; dmarc=pass (policy=none) header.from=mcusim.org Received: from localhost (unknown [193.164.254.100]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mcusim.org (Postfix) with ESMTPSA id A54DB6B665; Sun, 16 Apr 2023 09:39:41 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=mcusim.org; s=default; t=1681630781; bh=5M1rULaeKxLolxl3WNyEjQpNoTMK3VJwP2Razt4D2eM=; h=References:From:To:Cc:Subject:Date:In-reply-to; b=T3Js0gOUmuchHsCAwiW8fXX6BqJjqrzYcXL2gTEJN4/fNbNJ+qAPu7JjUWsOcyQ7T J7hRpg25eET52CjEWcsd/tYZM+yfdLbJ9QSHhRnmq2hdObAfDK/+P8UwfhWO6ybdKo 7yRbpA1z61LGxcqUB/uyIXNb0+HNe7BsrWvDFm8c= References: <86ildyucuv.fsf@peasant.tower.home> User-agent: mu4e 1.6.10; emacs 28.2 From: Dmitry Salychev To: Mitchell Horne Cc: freebsd-arm@freebsd.org Subject: Re: About PHYS_TO_DMAP Date: Sun, 16 Apr 2023 09:06:58 +0200 In-reply-to: Message-ID: <86zg78s2bj.fsf@peasant.tower.home> List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org MIME-Version: 1.0 Content-Type: text/plain X-Spamd-Result: default: False [-3.99 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.994]; DMARC_POLICY_ALLOW(-0.50)[mcusim.org,none]; R_DKIM_ALLOW(-0.20)[mcusim.org:s=default]; R_SPF_ALLOW(-0.20)[+a]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-arm@freebsd.org]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:36236, ipnet:2a00:dd80:3c::/48, country:US]; RCPT_COUNT_TWO(0.00)[2]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FREEFALL_USER(0.00)[dsl]; ARC_NA(0.00)[]; DKIM_TRACE(0.00)[mcusim.org:+]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_SOME(0.00)[]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4Pzhsf5SZpz4QhV X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N Mitchell Horne writes: > On 4/14/23 04:31, Dmitry Salychev wrote: >> Hi, >> I'm struggling to understand which KVA will be returned by >> PHYS_TO_DMAP >> on arm64. For example, if I'll create a DMA tag this way: >> bus_dma_tag_create( >> bus_get_dma_tag(dev), >> sc->buf_align, 0, /* alignment, boundary */ >> DMAP_MAX_PHYSADDR, /* low restricted addr */ >> DMAP_MIN_PHYSADDR, /* high restricted addr */ > > I think you are confused about the purpose of lowaddr and > highaddr. They specify the window of bus-space addresses that are > *inaccessible* *to the device* for DMA. It does not prevent you from > providing a buffer backed by physical memory within this range, but in > this case bounce pages will be used as an intermediate to perform the > DMA transaction. > > Most commonly, if the device can only address a 32-bit range, then the > values lowaddr=BUS_SPACE_MAXADDR_32BIT and highaddr=BUS_SPACE_MAXADDR > will be given. If the entire bus range is accessible to the device for > DMA, a zero-sized window will be given: lowaddr=BUS_SPACE_MAXADDR, > highaddr=BUS_SPACE_MAXADDR. > > This is all described in adequate detail in busdma(9), but it is still > not easily understood without a lot of code study, in my experience. > According to busdma(9), the window contains all addresses *greater than* lowaddr and *less than or equal to* highaddr, i.e. your example with the 32-bit range looks like: allowed prohibited ------------------------------(xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx> lowaddr highaddr BUS_SPACE_MAXADDR_32BIT BUS_SPACE_MAXADDR But my example looks a bit different: prohibited allowed prohibited xxxxxxxxxxxxxxxx]----------------------------(xxxxxxxxxxxxxxxxxxx> highaddr lowaddr DMAP_MIN_PHYSADDR DMAP_MAX_PHYSADDR I've found the only example of such DMA tag configuration in src/sys/dev/vnic/nicvf_queues.c at line 379 (rbdr_buff_dmat). >> NULL, NULL, /* filter, filterarg */ >> BUF_SIZE, 1, /* maxsize, nsegments */ >> BUF_SIZE, 0, /* maxsegsize, flags */ >> NULL, NULL, /* lockfunc, lockarg */ >> &dmat); >> in order to restrict any physical addresses but a window defined by >> DMAP_MIN_PHYSADDR and DMAP_MAX_PHYSADDR. Later on when I'll be >> mapping my mbuf (BUF_SIZE) with >> bus_dmamap_load_mbuf_sg(dmat, dmap, >> m, &segs, &nsegs, BUS_DMA_NOWAIT); >> I expect that >> m->m_data == PHYS_TO_DMAP(segs[0].ds_addr) > > Why do you expect or need this to be the case? > > busdma is not responsible for setting or modifying m_data, it comes > from wherever you allocated the mbuf. Calling > bus_dmamap_load_mbuf_sg() will prepare a DMA mapping where the m_data > buffer is used as the source/destination for the DMA transaction, but > busdma does not allocate the buffer itself. > I don't need this to be the case exactly, but I'd like to be able to access a "frame annotation" (64 bytes long) which is located exactly at the start of m_data buffer having that physical address is provided, i.e. fa = (struct dpaa2_fa *) mbuf->m_data; /* ... fa populated ... */ /* ... DMA transaction of the Tx frame (together with fa) ... */ /* ... Tx confirmation from HW (bus address of the frame only) ...*/ fa = (struct dpaa2_fa *) PHYS_TO_DMAP(paddr); >> but it isn't true. Could somebody explain what exactly is returned >> by >> PHYS_TO_DMAP in this case and whether it's possible to translate >> physical address to KVA as fast as possible (O(1) ideally). >> > > PHYS_TO_DMAP is always a linear calculation of: physaddr + DMAP_MIN_ADDRESS. > > I do not think PHYS_TO_DMAP is in use at all in this example, or > anywhere within busdma really. > Is there any other way to obtain KVA of a buffer mapped for DMA transaction by the physical address? I've been crawling source code for sometime already, but DMAP is the only thing I managed to find. >> Regards, >> Dmitry >> -- >> Open source software/hardware enthusiast >> hackaday.io/dsl | github.com/mcusim | patreon.com/salychev >> Regards, Dmitry -- Open source software/hardware enthusiast hackaday.io/dsl | github.com/mcusim | patreon.com/salychev