busdma dflt_lock on amd64 > 4 GB

[Jacques Caron]

Hi all,

Continuing on this story... [I took the liberty of CC'ing Scott and 
Soren], pr is amd64/87977 though it finally isn't amd64-specific 
but >4GB-specific.

There is really a big problem somewhere between ata and bus_dma for 
boxes with more than 4 GB RAM and more than 2 ata disks:
* bounce buffers will be needed
* ata will have bus_dma allocate bounce buffers:
hw.busdma.zone1.total_bpages: 32
hw.busdma.zone1.free_bpages: 32
hw.busdma.zone1.reserved_bpages: 0
hw.busdma.zone1.active_bpages: 0
hw.busdma.zone1.total_bounced: 27718
hw.busdma.zone1.total_deferred: 0
hw.busdma.zone1.lowaddr: 0xffffffff
hw.busdma.zone1.alignment: 2
hw.busdma.zone1.boundary: 65536

* if I do a dd with a bs=256000, 16 bounce pages will be used (most 
of the time). As long as I stay on the same disk, no more pages will be used.
* as soon as I access another disk (e.g. with another dd with the 
same bs=256000), another set of 16 pages will be used (bus_dma tags 
and maps are allocated on a per-channel basis), and all 32 bounce 
pages will be used (most of the time)
* and if I try to access a third disk, more bounce pages are needed and:
- one of ata_dmaalloc calls to bus_dma_tag_create has ALLOCNOW set
- busdma_machdep will not allocate more bounce pages in that case 
(the limit is imposed by maxsize in that situation, which has already 
been reached)
- ata_dmaalloc will fail
- but some other bus_dma_tag_create call without ALLOCNOW set will 
still cause bounce pages to be allocated, but deferred, and the 
non-existent lockfunc to be called, and panic.

Adding the standard lockfunc will (probably) solve the panic issue, 
but there will still be a problem with DMA in ata.

The same problems most probably exist with many other drivers.

I think we thus have two issues:
- providing a lockfunc in nearly all bus_dma_tag_create calls (or 
have a better default than a panic)
- allocating more bounce pages when needed in the ALLOCNOW case (with 
a logic similar to that used to allocate bounce pages in the non-ALLOCNOW case)

Thoughts?

Jacques.