ZFS Deadlock?

[Nick Sivo]

Hi,

One of our servers occasionally exhibits strange behavior under heavy
IO load. I think, based on the output from procstat -kk -a, it may be
a ZFS or VFS deadlock. Certain operations, including anything
involving the ZFS commands like zfs and zpool will hang. Running ls at
the root of a ZFS filesystem will also hang. Trying to access
snapshots in the .zfs/ folder will hang. None of these hung processes
can be killed. Eventually the machine will panic, if we don't reboot
it first, but that can take days after we start seeing this issue.
Strangely, our primary application (Hacker News) will keep running
without interruption until the panic.

Details of three occurrences can be found at
https://gist.github.com/kogir/acbd6d0e28ade0ee3aac

For the ones this month, it's on:
9.3-RELEASE-p10 FreeBSD 9.3-RELEASE-p10 #0: Tue Feb 24 21:28:03 UTC 2015

Those from October of last year were running an earlier 9.3 (exact
version unknown). The same hardware running 9.2 was solid for months
at a time. We never saw this issue on 9.2.

top output from the dying box right now:

last pid: 48083;  load averages:  0.24,  0.31,  0.27
120 processes: 1 running, 119 sleeping
CPU:  5.6% user,  0.0% nice,  1.7% system,  0.2% interrupt, 92.5% idle
Mem: 5722M Active, 249M Inact, 67G Wired, 352K Cache, 51G Free
ARC: 32G Total, 14G MFU, 8824M MRU, 52M Anon, 1800M Header, 7962M Other
Swap:

I'd show you the zpool configuration, but that would hang. We're not
using L2ARC or deduplication.

In any case, it's happening more frequently (twice this week), so I'd
like to get to the bottom of it if I can. Does this look like it could
be a filesystem issue? This will undoubtedly happen again. Is there
more information I should try to collect?

Thanks for your time and ideas/help you throw my way :)

Best,
Nick