ZFS Deadlock?

[Steven Hartland]

There are lots of fixes for ZFS issues in 10 so I'd recommend updated to 
10.1-RELEASE to see if you issue is already resolved there.

On 07/03/2015 03:13, Nick Sivo wrote:
> Hi,
>
> One of our servers occasionally exhibits strange behavior under heavy
> IO load. I think, based on the output from procstat -kk -a, it may be
> a ZFS or VFS deadlock. Certain operations, including anything
> involving the ZFS commands like zfs and zpool will hang. Running ls at
> the root of a ZFS filesystem will also hang. Trying to access
> snapshots in the .zfs/ folder will hang. None of these hung processes
> can be killed. Eventually the machine will panic, if we don't reboot
> it first, but that can take days after we start seeing this issue.
> Strangely, our primary application (Hacker News) will keep running
> without interruption until the panic.
>
> Details of three occurrences can be found at
> https://gist.github.com/kogir/acbd6d0e28ade0ee3aac
>
> For the ones this month, it's on:
> 9.3-RELEASE-p10 FreeBSD 9.3-RELEASE-p10 #0: Tue Feb 24 21:28:03 UTC 2015
>
> Those from October of last year were running an earlier 9.3 (exact
> version unknown). The same hardware running 9.2 was solid for months
> at a time. We never saw this issue on 9.2.
>
> top output from the dying box right now:
>
> last pid: 48083;  load averages:  0.24,  0.31,  0.27
> 120 processes: 1 running, 119 sleeping
> CPU:  5.6% user,  0.0% nice,  1.7% system,  0.2% interrupt, 92.5% idle
> Mem: 5722M Active, 249M Inact, 67G Wired, 352K Cache, 51G Free
> ARC: 32G Total, 14G MFU, 8824M MRU, 52M Anon, 1800M Header, 7962M Other
> Swap:
>
> I'd show you the zpool configuration, but that would hang. We're not
> using L2ARC or deduplication.
>
> In any case, it's happening more frequently (twice this week), so I'd
> like to get to the bottom of it if I can. Does this look like it could
> be a filesystem issue? This will undoubtedly happen again. Is there
> more information I should try to collect?
>
> Thanks for your time and ideas/help you throw my way :)
>
> Best,
> Nick
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"