ZFS deadlock on parallel ZFS operations FreeBSD 11.2 and 12.0

[Ole]

Hi,

I observed that FreeBSD Systems with ZFS will run into a deadlock if
there are many parallel zfs send/receive/snapshot processes.

I observed this on bare metal and virtual machines with FreeBSD 11.2
and 12.0. With RAM from 20 to 64G.

If the system is also on ZFS the whole system crashes. With only jails
on ZFS they freeze, but the Host system stays stable. But you can't
kill -9 the zfs processes. Only a poweroff stops the machine.

On a FreeBSD 12.0 VM (bhyve), 30G RAM, 5 CPUs, about 30 zfs operations,
mostly send and receive will crash the system.

There is no heavy load on the machine:

# top | head -8
last pid: 91503;  load averages:  0.34,  0.31,  0.29  up 0+22:50:47
11:24:00 536 processes: 1 running, 529 sleeping, 6 zombie
CPU:  0.9% user,  0.0% nice,  1.5% system,  0.2% interrupt, 97.4% idle
Mem: 165M Active, 872M Inact, 19G Wired, 264M Buf, 9309M Free
ARC: 11G Total, 2450M MFU, 7031M MRU, 216M Anon, 174M Header, 1029M
Other 8423M Compressed, 15G Uncompressed, 1.88:1 Ratio
Swap: 1024M Total, 1024M Free

I wonder if this is a BUG or normal behaviour. I could live with a
limited amount of parallel ZFS operation, but I don't want the whole
system to crash. 

Reducing the vfs.zfs.arc_max wont help.

Any Idea to handle with this?

regards
Ole
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: Digitale Signatur von OpenPGP
URL: <http://lists.freebsd.org/pipermail/freebsd-questions/attachments/20190215/37874c6c/attachment.sig>