AOC-USAS2-L8i zfs panics and SCSI errors in messages

Thu Oct 20 11:39:16 UTC 2011

Hi,

I´m in the process of vacating a Sun/Oracle system to a another Supermicro/FreeBSD system, doing zfs send/recv between. Two times now, the system has panicked while not doing anything at all, and it´s throwing alot of SCSI/CAM-related errors while doing IO-intensive operations, like send/recv, resilver, and zpool has sometimes reported read/write errors on the hard drives. Best part is that the errors in messages are about all hard drives at one time or another, and they are connected with separate cables, controllers and caddies. Specs:

HW:
1x  Supermicro X8SIL-F
2x  Supermicro AOC-USAS2-L8i
2x  Supermicro CSE-M35T-1B
1x  Intel Core i5 650 3,2GHz
4x  2GB 1333MHZ DDR3 ECC UDIMM
10x SAMSUNG HD204UI (in a raidz2 zpool)
1x  OCZ Vertex 3 240GB (L2ARC)

SW:
# uname -a
FreeBSD server 8.2-STABLE FreeBSD 8.2-STABLE #0: Mon Oct 10 09:12:25 UTC 2011     root at server:/usr/obj/usr/src/sys/GENERIC  amd64
# zpool get version pool1
NAME   PROPERTY  VALUE    SOURCE
pool1  version   28       default[/CODE]

I got the panic from the IPMI KVM:
http://i55.tinypic.com/synpzk.png

And an extract from /var/log/messages:
Oct 19 17:37:19 fs2-7 kernel: (da6:mps1:0:0:0): WRITE(10). CDB: 2a 0 6 13 66 f 0 0 f 0 
Oct 19 17:37:19 fs2-7 kernel: (da6:mps1:0:0:0): CAM status: SCSI Status Error
Oct 19 17:37:19 fs2-7 kernel: (da6:mps1:0:0:0): SCSI status: Check Condition
Oct 19 17:37:19 fs2-7 kernel: (da6:mps1:0:0:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Oct 19 17:37:19 fs2-7 kernel: (da6:mps1:0:0:0): WRITE(6). CDB: a 0 1 b2 2 0 
Oct 19 17:37:19 fs2-7 kernel: (da6:mps1:0:0:0): CAM status: SCSI Status Error
Oct 19 17:37:19 fs2-7 kernel: (da6:mps1:0:0:0): SCSI status: Check Condition
Oct 19 17:37:19 fs2-7 kernel: (da6:mps1:0:0:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Oct 19 17:40:38 fs2-7 kernel: (da9:mps1:0:4:0): SCSI command timeout on device handle 0x000c SMID 859
Oct 19 17:40:38 fs2-7 kernel: (da9:mps1:0:4:0): SCSI command timeout on device handle 0x000c SMID 495
Oct 19 17:40:38 fs2-7 kernel: (da9:mps1:0:4:0): SCSI command timeout on device handle 0x000c SMID 725
Oct 19 17:40:38 fs2-7 kernel: (da9:mps1:0:4:0): SCSI command timeout on device handle 0x000c SMID 722
Oct 19 17:40:38 fs2-7 kernel: (da9:mps1:0:4:0): SCSI command timeout on device handle 0x000c SMID 438
Oct 19 17:40:38 fs2-7 kernel: mps1: (1:4:0) terminated ioc 804b scsi 0 state c xfer 0
Oct 19 17:40:38 fs2-7 last message repeated 3 times
Oct 19 17:40:38 fs2-7 kernel: mps1: mpssas_abort_complete: abort request on handle 0x0c SMID 859 complete
Oct 19 17:40:38 fs2-7 kernel: mps1: mpssas_complete_tm_request: sending deferred task management request for handle 0x0c SMID 495
Oct 19 17:40:38 fs2-7 kernel: mps1: mpssas_abort_complete: abort request on handle 0x0c SMID 495 complete
Oct 19 17:40:38 fs2-7 kernel: mps1: mpssas_complete_tm_request: sending deferred task management request for handle 0x0c SMID 725
Oct 19 17:40:38 fs2-7 kernel: mps1: mpssas_abort_complete: abort request on handle 0x0c SMID 725 complete
Oct 19 17:40:38 fs2-7 kernel: mps1: mpssas_complete_tm_request: sending deferred task management request for handle 0x0c SMID 722
Oct 19 17:40:38 fs2-7 kernel: mps1: mpssas_abort_complete: abort request on handle 0x0c SMID 722 complete
Oct 19 17:40:38 fs2-7 kernel: mps1: mpssas_complete_tm_request: sending deferred task management request for handle 0x0c SMID 438
Oct 19 17:40:38 fs2-7 kernel: mps1: mpssas_abort_complete: abort request on handle 0x0c SMID 438 complete
Oct 19 17:40:38 fs2-7 kernel: (da9:mps1:0:4:0): WRITE(10). CDB: 2a 0 6 25 4f 75 0 0 b 0 
Oct 19 17:40:38 fs2-7 kernel: (da9:mps1:0:4:0): CAM status: SCSI Status Error
Oct 19 17:40:38 fs2-7 kernel: (da9:mps1:0:4:0): SCSI status: Check Condition
Oct 19 17:40:38 fs2-7 kernel: (da9:mps1:0:4:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Oct 19 17:40:38 fs2-7 kernel: (da9:mps1:0:4:0): WRITE(10). CDB: 2a 0 2d a5 10 ca 0 0 80 0 
Oct 19 17:40:38 fs2-7 kernel: (da9:mps1:0:4:0): CAM status: SCSI Status Error
Oct 19 17:40:38 fs2-7 kernel: (da9:mps1:0:4:0): SCSI status: Check Condition
Oct 19 17:40:38 fs2-7 kernel: (da9:mps1:0:4:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Oct 19 17:45:40 fs2-7 kernel: (da1:mps0:0:1:0): SCSI command timeout on device handle 0x000a SMID 976
Oct 19 17:45:41 fs2-7 kernel: (da1:mps0:0:1:0): SCSI command timeout on device handle 0x000a SMID 636
Oct 19 17:45:41 fs2-7 kernel: (da1:mps0:0:1:0): SCSI command timeout on device handle 0x000a SMID 888
Oct 19 17:45:41 fs2-7 kernel: (da1:mps0:0:1:0): SCSI command timeout on device handle 0x000a SMID 983
Oct 19 17:45:41 fs2-7 kernel: mps0: (0:1:0) terminated ioc 804b scsi 0 state c xfer 0
Oct 19 17:45:41 fs2-7 last message repeated 2 times
Oct 19 17:45:41 fs2-7 kernel: mps0: mpssas_abort_complete: abort request on handle 0x0a SMID 976 complete
Oct 19 17:45:41 fs2-7 kernel: mps0: mpssas_complete_tm_request: sending deferred task management request for handle 0x0a SMID 636
Oct 19 17:45:41 fs2-7 kernel: mps0: mpssas_abort_complete: abort request on handle 0x0a SMID 636 complete
Oct 19 17:45:41 fs2-7 kernel: mps0: mpssas_complete_tm_request: sending deferred task management request for handle 0x0a SMID 888
Oct 19 17:45:41 fs2-7 kernel: mps0: mpssas_abort_complete: abort request on handle 0x0a SMID 888 complete
Oct 19 17:45:41 fs2-7 kernel: mps0: mpssas_complete_tm_request: sending deferred task management request for handle 0x0a SMID 983
Oct 19 17:45:41 fs2-7 kernel: mps0: mpssas_abort_complete: abort request on handle 0x0a SMID 983 complete
Oct 19 17:45:41 fs2-7 kernel: (da1:mps0:0:1:0): WRITE(10). CDB: 2a 0 6 40 a7 2 0 0 3 0 
Oct 19 17:45:41 fs2-7 kernel: (da1:mps0:0:1:0): CAM status: SCSI Status Error
Oct 19 17:45:41 fs2-7 kernel: (da1:mps0:0:1:0): SCSI status: Check Condition
Oct 19 17:45:41 fs2-7 kernel: (da1:mps0:0:1:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Oct 19 17:45:42 fs2-7 kernel: (da1:mps0:0:1:0): WRITE(10). CDB: 2a 0 6 40 b0 9 0 0 9 0 
Oct 19 17:45:42 fs2-7 kernel: (da1:mps0:0:1:0): CAM status: SCSI Status Error
Oct 19 17:45:42 fs2-7 kernel: (da1:mps0:0:1:0): SCSI status: Check Condition
Oct 19 17:45:42 fs2-7 kernel: (da1:mps0:0:1:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)

What´s going on?

Regards
Karli Sjöberg