ZFS deadlock on parallel ZFS operations FreeBSD 11.2 and 12.0

[Ole]

Hi,

ok now I got a again unkillable ZFS process. It is only one 'zfs send'
command. Any Idea how to kill this process without powering off the
machine?

oot at jails1:/usr/home/admin # ps aux | grep 'zfs send'
root      17617   0.0  0.0  12944  3856  -  Is   Sat04       0:00.00 sudo zfs send -e -I cryptopool/iocage/jails/2fe7ae89-760e-423c-8e7f-4f504e0f08bf at 2019-
root      17618   0.0  0.0  12980  4036  -  D    Sat04       0:00.01 zfs send -e -I cryptopool/iocage/jails/2fe7ae89-760e-423c-8e7f-4f504e0f08bf at 2019-02-16
root      19299   0.0  0.0  11320  2588  3  S+   09:53       0:00.00 grep zfs send
root at jails1:/usr/home/admin # kill -9 17618
root at jails1:/usr/home/admin # ps aux | grep 'zfs send'
root      17617   0.0  0.0  12944  3856  -  Is   Sat04       0:00.00 sudo zfs send -e -I cryptopool/iocage/jails/2fe7ae89-760e-423c-8e7f-4f504e0f08bf at 2019-
root      17618   0.0  0.0  12980  4036  -  D    Sat04       0:00.01 zfs send -e -I cryptopool/iocage/jails/2fe7ae89-760e-423c-8e7f-4f504e0f08bf at 2019-02-16
root      19304   0.0  0.0  11320  2588  3  S+   09:53       0:00.00 grep zfs send

It is a FreeBSD 12.0 VM-Image running in a Bhyve VM. There is basicly only
py36-iocage installed, and there are 7 running Jails. 

There is 30G RAM and sysctl vfs.zfs.arc_max ist set to 20G. It seems that 
the whole zpool is in some kind of deadlock. All Jails are crashed, 
unkillable and I can not run any command inside. 

regards
Ole


Fri, 15 Feb 2019 11:34:23 +0100 - Ole <ole at free.de>:

> Hi,
> 
> I observed that FreeBSD Systems with ZFS will run into a deadlock if
> there are many parallel zfs send/receive/snapshot processes.
> 
> I observed this on bare metal and virtual machines with FreeBSD 11.2
> and 12.0. With RAM from 20 to 64G.
> 
> If the system is also on ZFS the whole system crashes. With only jails
> on ZFS they freeze, but the Host system stays stable. But you can't
> kill -9 the zfs processes. Only a poweroff stops the machine.
> 
> On a FreeBSD 12.0 VM (bhyve), 30G RAM, 5 CPUs, about 30 zfs
> operations, mostly send and receive will crash the system.
> 
> There is no heavy load on the machine:
> 
> # top | head -8
> last pid: 91503;  load averages:  0.34,  0.31,  0.29  up 0+22:50:47
> 11:24:00 536 processes: 1 running, 529 sleeping, 6 zombie
> CPU:  0.9% user,  0.0% nice,  1.5% system,  0.2% interrupt, 97.4% idle
> Mem: 165M Active, 872M Inact, 19G Wired, 264M Buf, 9309M Free
> ARC: 11G Total, 2450M MFU, 7031M MRU, 216M Anon, 174M Header, 1029M
> Other 8423M Compressed, 15G Uncompressed, 1.88:1 Ratio
> Swap: 1024M Total, 1024M Free
> 
> I wonder if this is a BUG or normal behaviour. I could live with a
> limited amount of parallel ZFS operation, but I don't want the whole
> system to crash. 
> 
> Reducing the vfs.zfs.arc_max wont help.
> 
> Any Idea to handle with this?
> 
> regards
> Ole
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: Digitale Signatur von OpenPGP
URL: <http://lists.freebsd.org/pipermail/freebsd-questions/attachments/20190219/73c59948/attachment.sig>