[Bug 237636] bhyve guest ZFS filesystem freezes in zcw->zcw_cv state

[bugzilla-noreply at freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237636

            Bug ID: 237636
           Summary: bhyve guest ZFS filesystem freezes in zcw->zcw_cv
                    state
           Product: Base System
           Version: 12.0-RELEASE
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Only Me
          Priority: ---
         Component: kern
          Assignee: bugs at FreeBSD.org
          Reporter: freebsd-bugs at morgandavis.net

I experience random freezes during file I/O that locks up the entire virtual
machine (except networking). Environment is FreeBSD 12.0-RELEASE bhyve guest
running with ZFS on a FreeBSD 12.0-RELEASE host also on ZFS.

During some specific disk operations, such as a sqlite database vacuum or git
'gc' operation, the VM will freeze up and cease performing disk operations. 
The VM can still respond to pings and accept network connections, but any disk
operation will just cause complete blocking.  The only remedy is to kill -9 the
bhyve host process and restart the guest.

Trying Control-T during one of these episodes always shows the kernel stuck in
zcw->zcw_cv.  For example, this "git gc" operation:

Enumerating objects: 12282, done.
Counting objects: 100% (12282/12282), done.
Delta compression using up to 4 threads
Compressing objects: 100% (12259/12259), done.
<--- system completely stops responding here, control-T entered -->
load: 0.47  cmd: git 90093 [zcw->zcw_cv] 31.31r 9.77u 0.09s 0% 123612k

This does not always happen and is not reproducible at will. However, the
freezes are occurring mostly with sqlite and occasionally with git.

Host system has 16GB RAM and guest has 5GB allocated.  The host does nothing
but run this one guest bhyve instance.

Guest's /boot/loader.conf disables ZFS prefetching:

vfs.zfs.prefetch_disable="1"

And after much searching for help on the freezing, I found this article which
suggested reducing ZFS arc max.
https://www.reddit.com/r/freebsd/comments/1u358s/freebsd_random_freeze_where_to_start/

So I added this to loader.conf, too:

vfs.zfs.arc_max="1G"

This occurs on FreeBSD 11 and now 12 as well.  I don't recall having this issue
under FreeBSD 10.

-- 
You are receiving this mail because:
You are the assignee for the bug.