kern/179118: [mfi] COMMAND 0x.. TIMEOUT AFTER ## SECONDS (Dell H710 Mini (blades))

[Ryan Steinmetz]

I attached a serial console to the box and got the following:

mfi0: COMMAND 0xffffff8002b91178 TIMEOUT AFTER 41 SECONDS
mfi0: I/O error, cmd=0xffffff8002b91178, error=0x3c
mfid0: hard error cmd=write 61548002-61548065
mfi0: g_vfs_done():COMMAND 0xffffff8002b91068 TIMEOUT AFTER 41 SECONDS
mfid0p2[WRITE(offset=31512494080, length=32768)]mfi0: error = 60
I/O error, cmd=0xffffff8002b91068, error=0x3c
mfid0: hard error cmd=write 184643042-184643105
mfi0: g_vfs_done():COMMAND 0xffffff8002b90760 TIMEOUT AFTER 41 SECONDS
mfid0p2[WRITE(offset=94537154560, length=32768)]mfi0: error = 60
I/O error, cmd=0xffffff8002b90760, error=0x3c
mfid0: hard error cmd=write 1500221282-1500221345
mfi0: g_vfs_done():COMMAND 0xffffff8002b90ed0 TIMEOUT AFTER 41 SECONDS
mfid0p2[WRITE(offset=768113213440, length=32768)]mfi0: error = 60
I/O error, cmd=0xffffff8002b90ed0, error=0x3c
mfid0: hard error cmd=write 1722048802-1722048865
mfi0: g_vfs_done():COMMAND 0xffffff8002b91288 TIMEOUT AFTER 41 SECONDS
mfid0p2[WRITE(offset=881688903680, length=32768)]mfi0: error = 60
I/O error, cmd=0xffffff8002b91288, error=0x3c
mfid0: hard error cmd=write 1722048866-1722048929
mfi0: g_vfs_done():COMMAND 0xffffff8002b91310 TIMEOUT AFTER 41 SECONDS
mfid0p2[WRITE(offset=881688936448, length=32768)]mfi0: error = 60
I/O error, cmd=0xffffff8002b91310, error=0x3c
mfid0: hard error cmd=write 1722049442-1722049505
mfi0: g_vfs_done():COMMAND 0xffffff8002b91398 TIMEOUT AFTER 41 SECONDS
mfid0p2[WRITE(offset=881689231360, length=32768)]mfi0: error = 60
I/O error, cmd=0xffffff8002b91398, error=0x3c
mfid0: hard error cmd=write 1722049506-1722049569
mfi0: g_vfs_done():COMMAND 0xffffff8002b915b8 TIMEOUT AFTER 41 SECONDS
mfid0p2[WRITE(offset=881689264128, length=32768)]mfi0: error = 60
I/O error, cmd=0xffffff8002b915b8, error=0x3c
mfid0: hard error cmd=write 1722050146-1722050209
g_vfs_done():mfid0p2[WRITE(offset=881689591808, length=32768)]error = 60
mfi0: COMMAND 0xffffff8002b914a8 TIMEOUT AFTER 40 SECONDS
mfi0: I/O error, cmd=0xffffff8002b914a8, error=0x3c
mfid0: hard error cmd=write 1722219234-1722219281
mfi0: g_vfs_done():COMMAND 0xffffff8002b916c8 TIMEOUT AFTER 30 SECONDS
mfid0p2[WRITE(offset=881776164864, length=24576)]mfi0: error = 60
I/O error, cmd=0xffffff8002b916c8, error=0x3c
mfid0: hard error cmd=write 1722093946-1722093953
g_vfs_done():mfid0p2[WRITE(offset=881712017408, length=4096)]error = 60
mfi0: cmd_tbolt 0xfffffe004a99a100 has invalid sync_cmd_idx=128 - skipping
mfi0: cmd_tbolt 0xfffffe004a99a200 has invalid sync_cmd_idx=128 - skipping
mfi0: cmd_tbolt 0xfffffe004a973e00 has invalid sync_cmd_idx=128 - skipping
mfi0: cmd_tbolt 0xfffffe004a99a380 has invalid sync_cmd_idx=128 - skipping
mfi0: cmd_tbolt 0xfffffe004a99a000 has invalid sync_cmd_idx=128 - skipping
mfi0: cmd_tbolt 0xfffffe004a97be00 has invalid sync_cmd_idx=128 - skipping
mfi0: cmd_tbolt 0xfffffe004a97bd80 has invalid sync_cmd_idx=128 - skipping
mfi0: cmd_tbolt 0xfffffe004a97bb80 has invalid sync_cmd_idx=128 - skipping
mfi0: cmd_tbolt 0xfffffe004a97bc80 has invalid sync_cmd_idx=128 - skipping
mfi0: cmd_tbolt 0xfffffe004a97ba80 has invalid sync_cmd_idx=128 - skipping
mfi0: COMMAND 0xffffff8002b91178 TIMEOUT AFTER 30 SECONDS
mfi0: I/O error, cmd=0xffffff8002b91178, error=0x3c
mfid0: hard error cmd=write 40610-40657
mfi0: g_vfs_done():COMMAND 0xffffff8002b91068 TIMEOUT AFTER 30 SECONDS
mfid0p2[WRITE(offset=20709376, length=24576)]mfi0: error = 60
I/O error, cmd=0xffffff8002b91068, error=0x3c
mfid0: hard error cmd=write 290-297
g_vfs_done():mfid0p2[WRITE(offset=65536, length=4096)]error = 60
/dev: got error 60 while accessing filesystem
panic: softdep_deallocate_dependencies: unrecovered I/O error
cpuid = 22
KDB: stack backtrace:
#0 0xffffffff80920c76 at kdb_backtrace+0x66
#1 0xffffffff808eac8e at panic+0x1ce
#2 0xffffffff80b09100 at clear_remove+0
#3 0xffffffff809670a0 at brelse+0x60
#4 0xffffffff80969a48 at bufdone+0x68
#5 0xffffffff808561b6 at g_io_schedule_up+0xa6
#6 0xffffffff8085673c at g_up_procbody+0x5c
#7 0xffffffff808bbdbf at fork_exit+0x11f
#8 0xffffffff80bc3a5e at fork_trampoline+0xe
Uptime: 2h0m19s

Dump failed. Partition too small.
mfi0: cmd_tbolt 0xfffffe004a99a100 has invalid sync_cmd_idx=128 - skipping
mfi0: cmd_tbolt 0xfffffe004a99a200 has invalid sync_cmd_idx=128 - skipping
Automatic reboot in 15 seconds - press a key on the console to abort
--> Press a key on the console to reboot,
--> or switch off the system now.
Rebooting...
cpu_reset: Restarting BSP
cpu_reset_proxy: Stopped CPU 22