[Bug 231117] I/O lockups inside bhyve vms

[bugzilla-noreply at freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=231117

            Bug ID: 231117
           Summary: I/O lockups inside bhyve vms
           Product: Base System
           Version: 11.2-RELEASE
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Only Me
          Priority: ---
         Component: kern
          Assignee: bugs at FreeBSD.org
          Reporter: niels at kristensen.io

We're running bhyve VMs on ~10-20 hypervisors.

Since upgrading everything to 11.2-RELEASE last week, we've seen i/o lockups
inside different VMs on multiple of these hypervisors. The VMs also run FreeBSD
11.2, and both the hypervisors and the VMs run ZFS for backup and quota
reasons.

We havn't had any issues on 11.1, but when we ran 11.0, we had some problems
with the ARC inside the VMs. The VM would be unable to make any disk i/o as we
experience now as well, but in 11.0 the problem could be "resolved" by
launching a memory hungry bin from tmpfs, causing ARC to release some memory,
and then we got working i/o again.

On 11.0 the affected VMs were generally i/o intensive VMs. Now we see random
lockups on different workloads.

The zvol timestamp on the hypervisors are not updated, so clearly writes fail
to go through.

Does anyone have some advise for debugging this issue further?

-- 
You are receiving this mail because:
You are the assignee for the bug.