SMP, the aac driver and command timeouts

[Aaron Wohl]

Yes it worked ok.  We got two servers both 2U SMP Xeon 2800mhz with
adaptec 5400S controllers from Lanny @ freebsdsystems.com.  The one thats
working ok was built from -current at Fri Aug  1 19:50:58 EDT 2003, its
in production hard to get any time on it to tinker.

The 2nd system (the one thats been crashing) is the same hardware plus a
scsi tape plugged into the scsi controller on the motherboard.
 Its been crashing lately with "command not in queue" but ive seeh the
 aac0 COMMAND ... TIMEOUT as well.

As for repeating it... the machine thats crashing tends to crash if I
dump and load a 1.5 gbyte mysql table, and or do amanda tape backups.  Im
sorry thats not much to go on.  Ive been try to get thru the weekly
amanda dump to tape each day this week but its crashing with "command not
in queue".  

Here is a stack backtrace:
gdb -k /usr/obj/usr/src/sys/PASODOBLE/kernel.debug vmcore.9
...
dmesg: kvm_read: 
---
Reading symbols from
/usr/obj/usr/src/sys/PASODOBLE/modules/usr/src/sys/modules/acpi/acpi.ko.debug...done.
Loaded symbols for
/usr/obj/usr/src/sys/PASODOBLE/modules/usr/src/sys/modules/acpi/acpi.ko.debug
Reading symbols from /boot/kernel/green_saver.ko...done.
Loaded symbols for /boot/kernel/green_saver.ko
#0  doadump () at /usr/src/sys/kern/kern_shutdown.c:240
240             dumping++;
(kgdb) where
#0  doadump () at /usr/src/sys/kern/kern_shutdown.c:240
#1  0xc0332b41 in boot (howto=256) at
/usr/src/sys/kern/kern_shutdown.c:372
#2  0xc0332f98 in panic () at /usr/src/sys/kern/kern_shutdown.c:550
#3  0xc01676b4 in aac_complete (context=0xcb918000, pending=1) at
/usr/src/sys/dev/aac/aacvar.h:535
#4  0xc03599ed in taskqueue_run (queue=0xc6768780) at
/usr/src/sys/kern/subr_taskqueue.c:205
#5  0xc0359ac3 in taskqueue_swi_run (dummy=0x0) at
/usr/src/sys/kern/subr_taskqueue.c:221
#6  0xc031c8d8 in ithread_loop (arg=0xc6768700) at
/usr/src/sys/kern/kern_intr.c:534
#7  0xc031b511 in fork_exit (callout=0xc031c700 <ithread_loop>, arg=0x0,
frame=0x0) at /usr/src/sys/kern/kern_fork.c:796
(kgdb) 

On Fri, 29 Aug 2003 08:15:37 -0600, "Scott Long" <scottl at freebsd.org>
said:
> Aaron Wohl wrote:
> 
> > Yeah im getting 2-3 aac driver related crashes a day now with -current on
> > a 5400s.
> > 
> > I was seeing that "aac0 ... COMMAND 0x...... TIMEOUT AFTER ... seconds"
> > as well.  I did a cvsup and rebuild/install yesterday.  Im not getting
> > that now but still geting "command not in queue" panics. from an adaptic
> > 5400S.
> 
> This is all quite serious.  Did the driver ever work for you?  Is this
> an SMP machine?  Is there a reproducable test case that I could use to
> debug it locally?
> 
> Scott
> 
>