Serious Dell Sadness - H200, H700, and H800

[Neil Schelly]

We've got some more information about the mpt testing we've been doing here.  The setup we're testing is Dell PowerEdge r610 servers with PERC H800 SAS/RAID cards connected to MD1200 shelves full of 12 SAS drives.  We've recreated the same problem on other configurations, including combinations of r510s, MD1220 shelves, PERC H700 cards, etc.  We've also eliminated any particular piece of hardware as faulty by running these on identical hardware configurations in mirrored setups on different physical pieces of hardware.  We've experienced these issues in FreeBSD 7.3, 8.1, and 8.2.  We've experienced this issue with either RAID10 logical drive configurations formatted with UFS or 6-disk JBOD configurations setup in a ZFS raidz volume.  We've triggered the problem with both bonnie++ and iozone.  All machines are runnning the latest firmware on the H700 and H800 cards.

The easiest method to reproduce this problem is with a ZFS configuration and using `iozone -a`.  We have a 6-disk raidz partition with a ZFS filesystem on it.  We just run `iozone -a` from within that filesystem, and I'd say 3 out of 4 times, it will eventually pause.  After 45-50 seconds of pausing, you'll start seeing the console and /var/log/messages output that looks something like: 
  mfi0: COMMAND 0xffffff8000db5fe0 TIMEOUT AFTER 105 SECONDS

If we let it go for a few days, it may actually "finish" and recover, but it's essentially just stuck and not recovering.  The system is responsive and fully operational except the dead controller at this point.  We cannot kill the iozone process that is hung on these IO operations, even with `kill -9`.  Like others have reported, we can run any of the mfiutil commands and the controller immediately begins to respond normally again.  Usually, the iozone test will complete, but sometimes it will even get stuck again on the same run.  

We compiled mfiutil with debugging symbols so we could run it with gdb and see exactly what was causing the controller to become responsive again.  It's the ioctl() call that does it.  For example:

`mfiutil show volumes` eventually gets to something like:
  mfi_dcmd_command (fd=7, opcode=50397184, buf=0x7fffffffe4a0, bufsize=1032, mbox=0x0, mboxlen=0, statusp=0x0)
  at /usr/src/usr.sbin/mfiutil/mfi_cmd.c:257
 * fd=7 is /dev/mfi0, where the command will be sent with an ioctl command
 * opcode=50397184 is the MFI_DCMD_LD_GET_LIST command

`mfiutil show battery` eventually gets to something like:
  mfi_dcmd_command (fd=7, opcode=84017152, buf=0x7fffffffea20, bufsize=48, mbox=0x0, mboxlen=0, statusp=0x7fffffffe9cf "")
  at /usr/src/usr.sbin/mfiutil/mfi_cmd.c:257
 * fd=7 is /dev/mfi0, where the command will be sent with an ioctl command
 * opcode=84017152 is the MFI_DCMD_BBU_GET_CAPACITY_INFO command

I wrote a small self-contained C program that can easily be modified to run any ioctl command you'd like and send it to /dev/mfi0 (attached).  Use it if you'd like at your own risk, but it's essentially just running an arbitrary command with ioctl, putting nothing into the memory range normally passed by the *buf pointer.  I did try sending random opcodes, and it didn't work, so it does have to be an opcode that the firmware will recognize at least, but it doesn't seem to matter which one.

I'm not sure where else we should be looking for a fix.  We can reliably reproduce the problem, analyze the system during the issue, and recover the system to a normal state.  If there's anyone who can help us troubleshoot this with any information we can gather or even a local login remotely accessible, we're open to ideas.  

--
Neil Schelly
Director of Uptime
Dynamic Network Services, Inc.
W: 603-296-1581
M: 508-410-4776
http://www.dyndns.com