How to map device addresses into user space

[Dr. Rolf Jansen]

Am 07.01.2013 um 17:21 schrieb John Baldwin:

> On Monday, January 07, 2013 12:25:33 PM Dr. Rolf Jansen wrote:
>> Am 07.01.2013 um 13:11 schrieb John Baldwin:
>>> On Monday, January 07, 2013 08:52:01 AM Dr. Rolf Jansen wrote:
>>>> Am 03.01.2013 um 14:45 schrieb Dr. Rolf Jansen:
>>>>> ...
>>>>> 
>>>>> So, how can I map device addresses into user space on FreeBSD?
>>>> 
>>>> Many Thanks to everybody, who responded.
>>>> 
>>>> I got it working. I wrote a PCI device driver, which in addition to
>>>> open/close/read/write responds also to ioctl and mmap requests from user
>>>> space. In its pci_attach routine, it assembles a custom address_space
>>>> struct containing the addresses of the two BAR's of my NI PCI-DAQ board
>>>> and two huge allocated memory regions for DMA...
>>> 
>>> Note that if your BARs are not contiguous you can allow mapping of things
>>> in the middle.  This is a potentital security hole if your driver can be
>>> opened by non-root.
>> 
>> I can understand this. Actually, on the present model of the DAQ board, the
>> BARs are contiguous, and in the present incarnation, my driver can only be
>> addressed by root. Anyway, I want to do it correctly, that means, I would
>> like to avoid right now potential holes bubbling up in the future, when
>> using my driver with other DAQ boards, and in different usage scenarios.
> 
> Note that the BARs being contiguous is a property of your BIOS or OS, not your 
> board. :)  The firmware/OS are free to assign whatever ranges to your BARs 
> that they wish.
> 
>>> A more robust approach is to define a virtual address space for
>>> your device. This is also a lot easier to program against in userland.
>>> For example, you could define offset 0 as mapping to BAR 0 and the next
>>> chunk
>> 
>>> of virtual address space (the offset) map to BAR 1, etc.:
>> I did this, but id didn't work out. The 2 BARs are contiguous, and the 2
>> DMA regions are also, but there is huge gap between BAR and DMA. And
>> depending on which mapping is done first, either the DMAs or the BARs
>> receive invalid paddr. So, I guess, defining a virtual address space is a
>> little bit more involved than simply getting the offsets straight as you
>> lined out.
>> 
>> I am still learning things, so please bear with me, if it is a dumb
>> question. How do I define a virtual address space for the device?
> 
> Ah, I'll explain a bit.  When you mmap a section of a file, the OS is treating 
> the file as if it were its own section of virtual memory, where files contents 
> were addressed by the offset.  That is, virtual address 0 of the file 
> corresponds to the data at file offset 0, and virtual address N corresponds to 
> the data at file offset N.
> 
> Now, think of your /dev/foo device as a file.  When you mmap() your /dev/foo, 
> the OS has assumes it has a similar virtual memory that is addressed by the 
> offset into the file.  However, for a character device, the OS needs the 
> driver to describe this virtual address space.  That is what you are doing 
> with d_mmap() and the 'offset' parameter.  You are telling it what physical 
> page backs each "virtual page".  It is important to note that when using 
> d_mmap(), these mappings are immutable.  Once you provide a mapping for a 
> given virtual page, the OS will cache it forever (or until reboot).
> 
> In other words, you can write your d_mmap() to give the appearance that your 
> /dev/foo "file" contains BAR 0 followed by BAR 1, etc.
> 
>>> static int
>>> mydaq_mmap(...)
>>> {
>>> 
>>> 	struct mydaq_softc *sc;  // you should be storing things like the
>>> 	pointers
>>> 	
>>>                            // to your BARs in a softc
>>> 	
>>> 	sc = dev->si_drv1;
>>> 	if (offset <= rman_get_size(sc->bar0res))
>>> 	
>>> 		return (rman_get_start(sc->bar0res) + offset);
>>> 	
>>> 	offset -= rman_get_size(sc->bar0res);
>>> 	if (offset <= rman_get_size(sc->bar1res))
>>> 	
>>> 		return (rman_get_start(sc->bar1res) + offset);
>>> 	
>>> 	offset -= rman_get_size(sc->bar1res);
>>> 	if (offset <= dmaAIBufSize)
>>> 	
>>> 		return (vtophys(sc->dma[0]) + offset);
>>> 	
>>> 	offset -= dmaAIBufSize;
>>> 	if (offset <= dmaAOBufSize)
>>> 	
>>> 		return (vtophys(sc->dma[1]) + offset);
>>> 	
>>> 	/* Invalid offset */
>>> 	return (-1);
>>> 
>>> }
>> 
>> I guess, you hacked this quickly together, and I picked the concept, so
>> this is fine. In order to make this working, the addresses should be
>> assigned to the *paddr parameter, and a status code should be returned. In
>> order to check things, of course I applied these minor corrections, but it
>> doesn't work for either the BAR block or the DMA block, depending on which
>> is mapped first.
> 
> Yes, it needs the changes you made.  Can you post your updated d_mmap()
> routine along with how you are mapping it in userland?  With what I posted
> above you would do something like this in userland:
> 
> const char *bar0, *bar1;
> 
> int fd = open("/dev/foo");
> 
> bar0 = mmap(NULL, <size of BAR 0>, PROT_READ, MAP_SHARED, fd, 0);
> bar1 = mmap(NULL, <size of BAR 1>, PROT_READ, MAP_SHARED, fd,
>    <size of BAR 0>);
> 
> etc.


Yes, now I understood it. I realized, that mydaq_mmap() is not called only once
for each mapping, but once for each page in each mapping. And finally I got it
working. Besides the corrections already mentioned, the comparison operators
needed to be changed from "<=" to "<". The working d_mmap() is now:


static int mydaq_mmap(struct cdev *dev, vm_ooffset_t offset, vm_paddr_t *paddr,
                      int nprot, vm_memattr_t *memattr)
{
   mydaq_pci_softc *sc = dev->si_drv1;

   if (offset < rman_get_size(sc->bar0res))
   {
      *paddr = rman_get_start(sc->bar0res) + offset;
      return 0;
   }

   offset -= rman_get_size(sc->bar0res);
   if (offset < rman_get_size(sc->bar1res))
   {
      *paddr = rman_get_start(sc->bar1res) + offset;
      return 0;
   }

   offset -= rman_get_size(sc->bar1res);
   if (offset < dmaAIBufferSize)
   {
      *paddr = vtophys(sc->dma[0]) + offset;
      return 0;
   }

   offset -= dmaAIBufferSize;
   if (offset < dmaAOBufferSize)
   {
      *paddr = vtophys(sc->dma[1]) + offset;
      return 0;
   }

   return -1;
}


Thank you very much again, for the big help.

Best regards

Rolf