SMP, the aac driver and command timeouts

[Aaron Wohl]

Yeah im getting 2-3 aac driver related crashes a day now with -current on
a 5400s.

I was seeing that "aac0 ... COMMAND 0x...... TIMEOUT AFTER ... seconds"
as well.  I did a cvsup and rebuild/install yesterday.  Im not getting
that now but still geting "command not in queue" panics. from an adaptic
5400S.
AAC0> controller details
Executing: controller details
Controller Information
----------------------
         Remote Computer: S
             Device Name: S
         Controller Type: Adaptec 5400S
             Access Mode: READ-WRITE
Controller Serial Number: Last Six Digits = 6B1825
         Number of Buses: 4
         Devices per Bus: 15
          Controller CPU: Strong Arm 110
    Controller CPU Speed: 233 Mhz
       Controller Memory: 144 Mbytes
           Battery State: Ok

Component Revisions
-------------------
                CLI: 1.0-0 (Build #5263)
                API: 1.0-0 (Build #5263)
    Miniport Driver: 1.0-0 (Build #5262)
Controller Software: 1.0-0 (Build #5262)
    Controller BIOS: 1.0-0 (Build #5262)
Controller Firmware: (Build #5262)
Controller Hardware: 3.3

AAC0>
uname -a (hostname edited)
FreeBSD xxx 5.1-CURRENT FreeBSD 5.1-CURRENT #34: Wed Aug 27 17:26:58 EDT
2003     xxx:/usr/obj/usr/src/sys/PASODOBLE  i386

*** email I sent this morning to the vendor we got our SMP hardware from
**
We are getting 2-3 crashes a day in the aac driver on the machine we
thought about replacing the processor on.  Ive read all the goings on on
the -current lists etc and trieed asking there about it.  The crashes
happen when doing heavy scsi io.  Either with disk intensive mysql jobs
or using the tape drive (amanda).  Each time the panic is "panic: command
not in queue" from the aac driver.  The other server we got from you is
not having these crashes.  But we havent updated the OS on it since Fri
Aug  1 19:50:58 EDT 2003.  Its interesting that the stack backtrace for
this crash ALWAYS has fork_exit in the stack backtrace.  Its trying to
remove a command from the response queue, but the item in the response
queue has a santity check that says which queue its on and its not listed
as being on the that queue.

I think you mentioned you where shipping 5.x on your server now?  Do you
get -current or is there a specific date/time for cvs checkout of the
operating system sources.

Id read the stuff on the -current list about having INVARIANTs on pissing
off the scsi driver due to new restrictions on doing INVARIANT checks
from drivers.  I tried building a kernel with INVARIANT off but it didnt
help.


panic: command not in queue
panic messages:
---
dmesg: kvm_read: 
---
Reading symbols from
/usr/obj/usr/src/sys/PASODOBLE/modules/usr/src/sys/modules/acpi/acpi.ko.debug...done.
Loaded symbols for
/usr/obj/usr/src/sys/PASODOBLE/modules/usr/src/sys/modules/acpi/acpi.ko.debug
Reading symbols from /boot/kernel/green_saver.ko...done.
Loaded symbols for /boot/kernel/green_saver.ko
#0  doadump () at /usr/src/sys/kern/kern_shutdown.c:240
240             dumping++;
(kgdb) where
#0  doadump () at /usr/src/sys/kern/kern_shutdown.c:240
#1  0xc0332b41 in boot (howto=256) at
/usr/src/sys/kern/kern_shutdown.c:372
#2  0xc0332f98 in panic () at /usr/src/sys/kern/kern_shutdown.c:550
#3  0xc01676b4 in aac_complete (context=0xcb918000, pending=1) at
/usr/src/sys/dev/aac/aacvar.h:535
#4  0xc03599ed in taskqueue_run (queue=0xc6768780) at
/usr/src/sys/kern/subr_taskqueue.c:205
#5  0xc0359ac3 in taskqueue_swi_run (dummy=0x0) at
/usr/src/sys/kern/subr_taskqueue.c:221
#6  0xc031c8d8 in ithread_loop (arg=0xc6768700) at
/usr/src/sys/kern/kern_intr.c:534
#7  0xc031b511 in fork_exit (callout=0xc031c700 <ithread_loop>, arg=0x0,
frame=0x0) at /usr/src/sys/kern/kern_fork.c:796
(kgdb)