Request for feedback on common data backstore in the kernel

[Hans Petter Selasky]

Hi,

Please keep me CC'ed, hence I'm not on all these lists.

In the kernel we currently have two different data backstores:

struct mbuf

and 

struct buf

These two backstores serve two different device types. "mbufs" are for network 
devices and "buf" is for disk devices.

Problem:

The current backstores are loaded into DMA by using the BUS-DMA framework. 
This appears not to be too fast according to Kip Macy. See:

http://perforce.freebsd.org/chv.cgi?CH=126455

Some ideas I have:

When a buffer is out out of range for a hardware device and a data-copy is 
needed I want to simply copy that data in smaller parts to/from a 
pre-allocated bounce buffer. I want to avoid allocating this buffer 
when "bus_dmamap_load()" is called.

For pre-allocated USB DMA memory I currently have:

struct usbd_page

struct usbd_page {
        void                    *buffer; // virtual address
        bus_size_t              physaddr; // as seen by one of my devices
        bus_dma_tag_t           tag;
        bus_dmamap_t            map;
        uint32_t                length;
};

Mostly only "length == PAGE_SIZE" is allowed. When USB allocates DMA memory it 
allocates the same size all the way and that is PAGE_SIZE bytes.

If two different PCI controllers want to communicate directly passing DMA 
buffers, technically one would need to translate the physical address for 
device 1 to the physical address as seen by device 2. If this translation 
table is sorted, the search will be rather quick. Another approach is to 
limit the number of translations:

#define N_MAX_PCI_TRANSLATE 4

struct usbd_page {
        void                    *buffer; // virtual address
        bus_size_t              physaddr[N_MAX_PCI_TRANSLATE];
        bus_dma_tag_t           tag;
        bus_dmamap_t            map;
        uint32_t                length;
};

Then PCI device 1 on bus X can use physaddr[0] and PCI device 2 on bus Y can 
use physaddr[1]. If the physaddr[] is equal to some magic then the DMA buffer 
is not reachable and must be bounced.

Then when two PCI devices talk together all they need to pass is a structure 
like this:

struct usbd_page_cache {
        struct usbd_page        *page_start;
        uint32_t                page_offset_buf;
        uint32_t                page_offset_end;
};

And the required DMA address is looked up in some nanos.

Has someone been thinking about this topic before ?

--HPS