busdma dflt_lock on amd64 > 4 GB

[Jacques Caron]

Hi Scott,

Thank you for your time on this issue...

At 18:29 26/10/2005, Scott Long wrote:
>I sent a long email on this on Dec 14, 2004.  I'll pull it up and 
>forward it out.  What I really should do is publish a definitive article
>on the whole topic.

Definitely! And add a link in the bus_dma man page, or expand its contents...

>As for 'limitations as to what can happen in the callback', there are
>none if you use the correct code structure.

Isn't the callback called from an interrupt context of some kind? (if 
you think I don't know a thing about kernel internals you've guessed 
right!) Would it be possible, for instance, to move this whole bunch of code:

         /* start ATAPI operation */
         if (ch->hw.command(request->device, ATA_PACKET_CMD, 0, 0, 
ATA_F_DMA)) {
             ata_prtdev(request->device, "error issuing ATAPI packet 
command\n");
             request->result = EIO;
             break;
         }

         /* wait for ready to write ATAPI command block */
         {
             int timeout = 5000; /* might be less for fast devices */
             while (timeout--) {
                 int reason = ATA_IDX_INB(ch, ATA_IREASON);
                 int status = ATA_IDX_INB(ch, ATA_STATUS);

                 if (((reason & (ATA_I_CMD | ATA_I_IN)) |
                      (status & (ATA_S_DRQ | ATA_S_BUSY))) == ATAPI_P_CMDOUT)
                     break;
                 DELAY(20);
             }
             if (timeout <= 0) {
                 ata_prtdev(request->device,"timeout waiting for 
ATAPI ready\n");
                 request->result = EIO;
                 break;
             }
         }

         /* this seems to be needed for some (slow) devices */
         DELAY(10);

         /* output actual command block */
         ATA_IDX_OUTSW_STRM(ch, ATA_DATA,
                            (int16_t *)request->u.atapi.ccb,
                            (request->device->param->config & 
ATA_PROTO_MASK) ==
                            ATA_PROTO_ATAPI_12 ? 6 : 8);

         /* start DMA engine */
         if (ch->dma->start(ch)) {
             request->result = EIO;
             break;
         }

into a callback? It's mainly all the DELAY's that seem out of place 
in a callback to me...

Note for Soren: there is a good explanation at the end of 
http://freebsd.active-venture.com/arch-handbook/isa-driver-busmem.html 
of callback integration strategies...

>>>No.  Some tags specifically should not permit deferals.
>>
>>How do they do that? Setting BUS_DMA_ALLOCNOW in the tag, or 
>>BUS_DMA_NOWAIT in the map_load, or both, or something else?
>
>They set it by using NULL as the lockfunc.

Errrr... I'm lost here. I understood that you should use NULL as the 
lockfunc if and only if you know there won't be any deferrals (and I 
know by experience now that not having a lockfunc and having a 
deferral cause a panic!). So having NULL as the lockfunc is a 
consequence of not having any deferrals, it won't prevent them.

>Actually, one map per tag is not common.  If the ATA driver supported
>tagged queuing (which I assume that it will someday for SATAII, yes?)

I'm waiting for it eagerly, I have quite a bunch of hardware all set for it :-)

>If a map is being created for every drive in the system, and the result
>is that not enough bounce pages are being reserved for all three drives
>to operate concurrently, then there might be a bug in busdma.  We should
>discuss this offline.

Maps are created for every busy drive in the system, but I believe 
they each belong to a separate tag (all tags and maps are created on 
the fly for each I/O operation, as I understand it). If three drives 
are (really) busy at the same time, a panic is guaranteed if bounce 
buffers are needed. I'm trying to figure out since yesterday where 
the problem is (busdma or ata), but I think it's "somewhere in 
between", with each part assuming that the other does things some way 
or another.

>>In this case 32 bounce pages (out of 8 GB RAM) for 6 disks seems 
>>like a very tight bottleneck to me.
>
>If that's all that is needed to saturate non-tagged ATA, then there is
>nothing wrong with that.

It might be enough to saturate one or two non-tagged ATA drives, but 
certainly not 6... Anyway, right now, it's not even saturation I have 
to deal with, but just a brutal panic :-(

Jacques.