From nobody Wed May 03 10:34:59 2023 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QBDZr6vszz49DQJ for ; Wed, 3 May 2023 11:03:32 +0000 (UTC) (envelope-from dsl@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QBDZr6P25z3jCV; Wed, 3 May 2023 11:03:32 +0000 (UTC) (envelope-from dsl@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1683111812; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=UM7ISAe2X+6BBR2ulf+b/7lzolNSvAFiWah2tt5wQGY=; b=lPql3ye38tz91jW+SzfxpBVautGrOggk6BGYxBYycxcOWDRNpXZK+9GXW5imiwp4YEPP8T VXp73gOMuNUV1x28caYMEmAw4VHlcUjCCSqH75Exl11SrOZeaGr3nyShFrC1rbfr5L380q ttUPrJgARxXAAyv2sKpLJ3ynHjy2V4xLjz040HDSYKAvRJb8T7ochVOhisl6w4o8FwFBvl jRXbRfVlZt2D48zyOOh29rJjL4dfB7vq/LZ6I4v01hE/9xikw+5ymNETOyfC7oJHqi5o4p zQPXLbRAc+7FrMQVZLYCyYpe+Y/Vn81bVjZ/NDPUPsUJ4SJPCNZENKMrZjPE/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1683111812; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=UM7ISAe2X+6BBR2ulf+b/7lzolNSvAFiWah2tt5wQGY=; b=fZ1oM6fN8O4zPOFtFY/EoG6lELXsLNvlGvq+RaNxusMzwO+OGhicwEjceCJruolX9LwbxZ sWTuMe5d9+CT+EHhZWpYHAANVkLu/cttAR+e76CF/gJ9BKjJifVMpMeJlkL/S/TOonhFqz mNoTAhKHmPYrN1fiCxgoBeJ83+QuDOnSrAhRfMhmw+JADRdueUv4ahWL1K6xGc0+4oq5mG P3fr0amxKuQjLLPZGm9VaC2HgWDAEE8bUvZ7zd2t5H0PSK+woGp35eRDvER5D00nUIa5Hq uKsqVYW0aEWjrCuPa1+qpJhlLkB/00y3BFsGS6PxlHw1B3nOLLD12jQNHl7C5A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1683111812; a=rsa-sha256; cv=none; b=eMA1XyuR/uu3lmusLoPvUR5QrO0btwUi93GjE94s4a7bPU92rJE1mvzUMFwklMm/OdmIIT JX9FRbVltL+UcEX81hsIs/C7VHDYPL0CX59E3Yac2CebKDw6Yf/OjE6F+GUaYEVHQiyEs9 B18NCAJIMwXKkTE82lR7L2DOohrCidM8iEzpoLSGSE02X7iyobUFGqIYJYMS/J74qlnoz2 C3tmg/4XRzkfLKUrG+N756sWIiczS0vzfC1cTCuonaPCYa3bv9v8MO3kNQmtxH4iObygRW dxKhJIqOOwnkqzCitRpbslQLZRxXK6HOp3ltF3Mjqp6cRwJlInNMigSwNzivmA== Received: from localhost (unknown [193.164.254.99]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: dsl) by smtp.freebsd.org (Postfix) with ESMTPSA id 4QBDZr2F8rzfvr; Wed, 3 May 2023 11:03:32 +0000 (UTC) (envelope-from dsl@freebsd.org) References: <86ildyucuv.fsf@peasant.tower.home> <86zg78s2bj.fsf@peasant.tower.home> <08ff62fd-89db-d429-9ef3-04f213118f83@freebsd.org> User-agent: mu4e 1.6.10; emacs 28.2 From: Dmitry Salychev To: Mitchell Horne Cc: freebsd-arm@freebsd.org Subject: Re: About PHYS_TO_DMAP Date: Wed, 03 May 2023 12:34:59 +0200 In-reply-to: <08ff62fd-89db-d429-9ef3-04f213118f83@freebsd.org> Message-ID: <86r0rxllrw.fsf@peasant.tower.home> List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org MIME-Version: 1.0 Content-Type: text/plain X-ThisMailContainsUnwantedMimeParts: N Mitchell Horne writes: > On 4/16/23 04:06, Dmitry Salychev wrote: >> Mitchell Horne writes: >> >>> On 4/14/23 04:31, Dmitry Salychev wrote: >>>> Hi, >>>> I'm struggling to understand which KVA will be returned by >>>> PHYS_TO_DMAP >>>> on arm64. For example, if I'll create a DMA tag this way: >>>> bus_dma_tag_create( >>>> bus_get_dma_tag(dev), >>>> sc->buf_align, 0, /* alignment, boundary */ >>>> DMAP_MAX_PHYSADDR, /* low restricted addr */ >>>> DMAP_MIN_PHYSADDR, /* high restricted addr */ >>> >>> I think you are confused about the purpose of lowaddr and >>> highaddr. They specify the window of bus-space addresses that are >>> *inaccessible* *to the device* for DMA. It does not prevent you from >>> providing a buffer backed by physical memory within this range, but in >>> this case bounce pages will be used as an intermediate to perform the >>> DMA transaction. >>> >>> Most commonly, if the device can only address a 32-bit range, then the >>> values lowaddr=BUS_SPACE_MAXADDR_32BIT and highaddr=BUS_SPACE_MAXADDR >>> will be given. If the entire bus range is accessible to the device for >>> DMA, a zero-sized window will be given: lowaddr=BUS_SPACE_MAXADDR, >>> highaddr=BUS_SPACE_MAXADDR. >>> >>> This is all described in adequate detail in busdma(9), but it is still >>> not easily understood without a lot of code study, in my experience. >>> >> According to busdma(9), the window contains all addresses *greater >> than* >> lowaddr and *less than or equal to* highaddr, i.e. your example with the >> 32-bit range looks like: >> allowed prohibited >> ------------------------------(xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx> >> lowaddr highaddr >> BUS_SPACE_MAXADDR_32BIT BUS_SPACE_MAXADDR >> But my example looks a bit different: >> prohibited allowed prohibited >> xxxxxxxxxxxxxxxx]----------------------------(xxxxxxxxxxxxxxxxxxx> >> highaddr lowaddr >> DMAP_MIN_PHYSADDR DMAP_MAX_PHYSADDR >> > > Okay I see, you are using MAX as lowaddr and MIN as highaddr. I read > it backwards the first time. > > However, I looked at the code and I'm not sure that it will handle the > case where lowaddr > highaddr correctly. Just a warning to you. > >> I've found the only example of such DMA tag configuration in >> src/sys/dev/vnic/nicvf_queues.c at line 379 (rbdr_buff_dmat). >> > I think this example is misleading, if not incorrect. All DMA > transactions for this tag will have a destination (physical memory > address) that falls within the [DMAP_MIN_PHYSADDR,DMAP_MAX_PHYSADDR] > range. So, the specified boundaries are not meaningful, and don't > represent a real constraint of the device. At least, this is my > understanding of it. I don't know the bus_dma implementation details yet unfortunately and thought that DMA tag created with [DMAP_MIN_PHYSADDR,DMAP_MAX_PHYSADDR] window is required to calculate KVA by PHYS_TO_DMAP properly. However, PHYS_TO_DMAP gave me an address to access frame annotation even with DMA tag created without a restriction window at all, i.e. BUS_SPACE_MAXADDR used for both lowaddr and highaddr. I've opened a review https://reviews.freebsd.org/D39946 if you'll be interested in details. > >> >>>> NULL, NULL, /* filter, filterarg */ >>>> BUF_SIZE, 1, /* maxsize, nsegments */ >>>> BUF_SIZE, 0, /* maxsegsize, flags */ >>>> NULL, NULL, /* lockfunc, lockarg */ >>>> &dmat); >>>> in order to restrict any physical addresses but a window defined by >>>> DMAP_MIN_PHYSADDR and DMAP_MAX_PHYSADDR. Later on when I'll be >>>> mapping my mbuf (BUF_SIZE) with >>>> bus_dmamap_load_mbuf_sg(dmat, dmap, >>>> m, &segs, &nsegs, BUS_DMA_NOWAIT); >>>> I expect that >>>> m->m_data == PHYS_TO_DMAP(segs[0].ds_addr) >>> >>> Why do you expect or need this to be the case? >>> >>> busdma is not responsible for setting or modifying m_data, it comes >>> from wherever you allocated the mbuf. Calling >>> bus_dmamap_load_mbuf_sg() will prepare a DMA mapping where the m_data >>> buffer is used as the source/destination for the DMA transaction, but >>> busdma does not allocate the buffer itself. >>> >> I don't need this to be the case exactly, but I'd like to be able to >> access a "frame annotation" (64 bytes long) which is located exactly at >> the start of m_data buffer having that physical address is provided, i.e. >> fa = (struct dpaa2_fa *) mbuf->m_data; >> /* ... fa populated ... */ >> /* ... DMA transaction of the Tx frame (together with fa) ... */ >> /* ... Tx confirmation from HW (bus address of the frame >> only) ...*/ >> fa = (struct dpaa2_fa *) PHYS_TO_DMAP(paddr); >> > > The short answer is that you are using PHYS_TO_DMAP correctly. Every > real paddr should have a corresponding VA in the DMAP that is valid > for read/write access to that memory. However, it is unusual to see > DMAP usage in driver code. > > It is hard for me to say exactly what you should be doing without more > information. > > In general, the m_data pointer remains valid after the DMA > transaction. If your updated frame annotation is located at the > beginning of this buffer, then you want to use the same m_data pointer > to access it. > > I guess the challenge is that the TX confirmation occurs in a separate > context (interrupt handler?) where the mbuf reference is lost? I > believe the usual practice is to maintain some kind of ring or queue > of transmitted mbuf+dma_map_t pairs, which can be handled together in > the TX interrupt handler (unload DMA map, free and/or reassign mbuf). This is correct. However, HW (firmware in my case) provides a frame descriptor which carries physical address of the Rx/Tx buffer only. This is why I needed a way to access the buffer itself from the kernel address space. > > Of course, you must use appropriate sync calls before accessing the > buffer again on the host, probably: bus_dmamap_sync(tag, map, > BUS_DMASYNC_POSTREAD | BUS_DMASYNC_POSTWRITE). I'm sure you know this. > >>>> but it isn't true. Could somebody explain what exactly is returned >>>> by >>>> PHYS_TO_DMAP in this case and whether it's possible to translate >>>> physical address to KVA as fast as possible (O(1) ideally). >>>> >>> >>> PHYS_TO_DMAP is always a linear calculation of: physaddr + DMAP_MIN_ADDRESS. >>> >>> I do not think PHYS_TO_DMAP is in use at all in this example, or >>> anywhere within busdma really. >>> >> Is there any other way to obtain KVA of a buffer mapped for DMA >> transaction by the physical address? I've been crawling source code for >> sometime already, but DMAP is the only thing I managed to find. >> >>> Regards, >>>> Dmitry >>>> -- >>>> Open source software/hardware enthusiast >>>> hackaday.io/dsl | github.com/mcusim | patreon.com/salychev >>>> >> Regards, >> Dmitry -- Open source software/hardware enthusiast hackaday.io/dsl | github.com/mcusim | patreon.com/salychev