ZFS deadlock with virtio (was: ZFS deadlock on parallel ZFS operations FreeBSD 11.2 and 12.0)
Ole
ole at free.de
Mon Mar 4 09:44:36 UTC 2019
Hello,
I have done some investigations. I think that there a two different
problems, so lets focus on the bhyve VM. I can now reproduce the
behaviour very well. It seems to be connected to the virtio disks.
The disk stack is:
Geli-encryption
Zpool (mirror)
Zvol
virtio
Zpool
- Hostsystem is FreeBSD 11.2
- VM is FreeBSD 12.0 (VM-Raw image + additional disk for zpool)
- VM is controlled by vm-bhyve
- inside the VM there are 5 to 10 running jails (managed with iocage)
If I start the Bhyve VM and let the Backups run (~10 operations per
hour) the Zpool inside the VM will crash after 1 to 2 days.
If I change the Disk from irtio-blk to ahci-hd, the VM keeps stable.
regards
Ole
Tue, 19 Feb 2019 10:17:17 +0100 - Ole <ole at free.de>:
> Hi,
>
> ok now I got a again unkillable ZFS process. It is only one 'zfs send'
> command. Any Idea how to kill this process without powering off the
> machine?
>
> oot at jails1:/usr/home/admin # ps aux | grep 'zfs send'
> root 17617 0.0 0.0 12944 3856 - Is Sat04 0:00.00
> sudo zfs send -e -I
> cryptopool/iocage/jails/2fe7ae89-760e-423c-8e7f-4f504e0f08bf at 2019-
> root 17618 0.0 0.0 12980 4036 - D Sat04 0:00.01
> zfs send -e -I
> cryptopool/iocage/jails/2fe7ae89-760e-423c-8e7f-4f504e0f08bf at 2019-02-16
> root 19299 0.0 0.0 11320 2588 3 S+ 09:53 0:00.00
> grep zfs send root at jails1:/usr/home/admin # kill -9 17618
> root at jails1:/usr/home/admin # ps aux | grep 'zfs send' root
> 17617 0.0 0.0 12944 3856 - Is Sat04 0:00.00 sudo zfs
> send -e -I
> cryptopool/iocage/jails/2fe7ae89-760e-423c-8e7f-4f504e0f08bf at 2019-
> root 17618 0.0 0.0 12980 4036 - D Sat04 0:00.01
> zfs send -e -I
> cryptopool/iocage/jails/2fe7ae89-760e-423c-8e7f-4f504e0f08bf at 2019-02-16
> root 19304 0.0 0.0 11320 2588 3 S+ 09:53 0:00.00
> grep zfs send
>
> It is a FreeBSD 12.0 VM-Image running in a Bhyve VM. There is basicly
> only py36-iocage installed, and there are 7 running Jails.
>
> There is 30G RAM and sysctl vfs.zfs.arc_max ist set to 20G. It seems
> that the whole zpool is in some kind of deadlock. All Jails are
> crashed, unkillable and I can not run any command inside.
>
> regards
> Ole
>
>
> Fri, 15 Feb 2019 11:34:23 +0100 - Ole <ole at free.de>:
>
> > Hi,
> >
> > I observed that FreeBSD Systems with ZFS will run into a deadlock if
> > there are many parallel zfs send/receive/snapshot processes.
> >
> > I observed this on bare metal and virtual machines with FreeBSD 11.2
> > and 12.0. With RAM from 20 to 64G.
> >
> > If the system is also on ZFS the whole system crashes. With only
> > jails on ZFS they freeze, but the Host system stays stable. But you
> > can't kill -9 the zfs processes. Only a poweroff stops the machine.
> >
> > On a FreeBSD 12.0 VM (bhyve), 30G RAM, 5 CPUs, about 30 zfs
> > operations, mostly send and receive will crash the system.
> >
> > There is no heavy load on the machine:
> >
> > # top | head -8
> > last pid: 91503; load averages: 0.34, 0.31, 0.29 up 0+22:50:47
> > 11:24:00 536 processes: 1 running, 529 sleeping, 6 zombie
> > CPU: 0.9% user, 0.0% nice, 1.5% system, 0.2% interrupt, 97.4%
> > idle Mem: 165M Active, 872M Inact, 19G Wired, 264M Buf, 9309M Free
> > ARC: 11G Total, 2450M MFU, 7031M MRU, 216M Anon, 174M Header, 1029M
> > Other 8423M Compressed, 15G Uncompressed, 1.88:1 Ratio
> > Swap: 1024M Total, 1024M Free
> >
> > I wonder if this is a BUG or normal behaviour. I could live with a
> > limited amount of parallel ZFS operation, but I don't want the whole
> > system to crash.
> >
> > Reducing the vfs.zfs.arc_max wont help.
> >
> > Any Idea to handle with this?
> >
> > regards
> > Ole
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: Digitale Signatur von OpenPGP
URL: <http://lists.freebsd.org/pipermail/freebsd-questions/attachments/20190304/809f93ca/attachment.sig>
More information about the freebsd-questions
mailing list