ZFS deadlock with virtio (was: ZFS deadlock on parallel ZFS operations FreeBSD 11.2 and 12.0)

Ole ole at free.de
Mon Mar 4 09:44:36 UTC 2019


Hello,

I have done some investigations. I think that there a two different
problems, so lets focus on the bhyve VM. I can now reproduce the
behaviour very well. It seems to be connected to the virtio disks.

The disk stack is:

Geli-encryption
Zpool (mirror)
Zvol
virtio
Zpool

- Hostsystem is FreeBSD 11.2
- VM is FreeBSD 12.0 (VM-Raw image + additional disk for zpool)
- VM is controlled by vm-bhyve
- inside the VM there are 5 to 10 running jails (managed with iocage)

If I start the Bhyve VM and let the Backups run (~10 operations per
hour) the Zpool inside the VM will crash after 1 to 2 days.

If I change the Disk from irtio-blk to ahci-hd, the VM keeps stable.

regards
Ole

Tue, 19 Feb 2019 10:17:17 +0100 - Ole <ole at free.de>:

> Hi,
> 
> ok now I got a again unkillable ZFS process. It is only one 'zfs send'
> command. Any Idea how to kill this process without powering off the
> machine?
> 
> oot at jails1:/usr/home/admin # ps aux | grep 'zfs send'
> root      17617   0.0  0.0  12944  3856  -  Is   Sat04       0:00.00
> sudo zfs send -e -I
> cryptopool/iocage/jails/2fe7ae89-760e-423c-8e7f-4f504e0f08bf at 2019-
> root      17618   0.0  0.0  12980  4036  -  D    Sat04       0:00.01
> zfs send -e -I
> cryptopool/iocage/jails/2fe7ae89-760e-423c-8e7f-4f504e0f08bf at 2019-02-16
> root      19299   0.0  0.0  11320  2588  3  S+   09:53       0:00.00
> grep zfs send root at jails1:/usr/home/admin # kill -9 17618
> root at jails1:/usr/home/admin # ps aux | grep 'zfs send' root
> 17617   0.0  0.0  12944  3856  -  Is   Sat04       0:00.00 sudo zfs
> send -e -I
> cryptopool/iocage/jails/2fe7ae89-760e-423c-8e7f-4f504e0f08bf at 2019-
> root      17618   0.0  0.0  12980  4036  -  D    Sat04       0:00.01
> zfs send -e -I
> cryptopool/iocage/jails/2fe7ae89-760e-423c-8e7f-4f504e0f08bf at 2019-02-16
> root      19304   0.0  0.0  11320  2588  3  S+   09:53       0:00.00
> grep zfs send
> 
> It is a FreeBSD 12.0 VM-Image running in a Bhyve VM. There is basicly
> only py36-iocage installed, and there are 7 running Jails. 
> 
> There is 30G RAM and sysctl vfs.zfs.arc_max ist set to 20G. It seems
> that the whole zpool is in some kind of deadlock. All Jails are
> crashed, unkillable and I can not run any command inside. 
> 
> regards
> Ole
> 
> 
> Fri, 15 Feb 2019 11:34:23 +0100 - Ole <ole at free.de>:
> 
> > Hi,
> > 
> > I observed that FreeBSD Systems with ZFS will run into a deadlock if
> > there are many parallel zfs send/receive/snapshot processes.
> > 
> > I observed this on bare metal and virtual machines with FreeBSD 11.2
> > and 12.0. With RAM from 20 to 64G.
> > 
> > If the system is also on ZFS the whole system crashes. With only
> > jails on ZFS they freeze, but the Host system stays stable. But you
> > can't kill -9 the zfs processes. Only a poweroff stops the machine.
> > 
> > On a FreeBSD 12.0 VM (bhyve), 30G RAM, 5 CPUs, about 30 zfs
> > operations, mostly send and receive will crash the system.
> > 
> > There is no heavy load on the machine:
> > 
> > # top | head -8
> > last pid: 91503;  load averages:  0.34,  0.31,  0.29  up 0+22:50:47
> > 11:24:00 536 processes: 1 running, 529 sleeping, 6 zombie
> > CPU:  0.9% user,  0.0% nice,  1.5% system,  0.2% interrupt, 97.4%
> > idle Mem: 165M Active, 872M Inact, 19G Wired, 264M Buf, 9309M Free
> > ARC: 11G Total, 2450M MFU, 7031M MRU, 216M Anon, 174M Header, 1029M
> > Other 8423M Compressed, 15G Uncompressed, 1.88:1 Ratio
> > Swap: 1024M Total, 1024M Free
> > 
> > I wonder if this is a BUG or normal behaviour. I could live with a
> > limited amount of parallel ZFS operation, but I don't want the whole
> > system to crash. 
> > 
> > Reducing the vfs.zfs.arc_max wont help.
> > 
> > Any Idea to handle with this?
> > 
> > regards
> > Ole  
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: Digitale Signatur von OpenPGP
URL: <http://lists.freebsd.org/pipermail/freebsd-questions/attachments/20190304/809f93ca/attachment.sig>


More information about the freebsd-questions mailing list