[Bug 264417] bhyve guest cannot be restarted if it has usb xhci pci controller passed thru to it: /usr/src/usr.sbin/bhyve, L:451

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=264417

            Bug ID: 264417
           Summary: bhyve guest cannot be restarted if it has usb xhci pci
                    controller passed thru to it: /usr/src/usr.sbin/bhyve,
                    L:451
           Product: Base System
           Version: 13.1-RELEASE
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Only Me
          Priority: ---
         Component: bhyve
          Assignee: virtualization@FreeBSD.org
          Reporter: lettergrabber@yandex.ru

I upgraded FreeBSD 13.0-RELEASE to FreeBSD 13.1-RELEASE and now I can't restart
VM.

What I doing:

I start VM with the command:
sudo bhyve -c 8,sockets=2,cores=2,threads=2 -m 16G -H -w -S \
  -s 0,hostbridge \
  -s 3,ahci-cd,/neat/iso/Win10_21H1_Russian_x64.iso \
  -s 4,ahci-hd,/neat/bsd1/vms/w10.img \
  -s 5,virtio-net,tap0 \
  -s 7,passthru,216/0/0 \
  -s 29,fbuf,tcp=0.0.0.0:5900,w=1920,h=1200,wait \
  -s 30,xhci,tablet \
  -s 31,lpc \
  -l com1,stdio \
  -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
  w10

Then, I connect to it:
vncviewer freebsd:0

I do some work inside VM, include that which uses 'usb xhci pci' I passed thru
to VM, there it is:
xhci3@pci0:216:0:0: class=0x0c0330 rev=0x03 hdr=0x00 vendor=0x1912
device=0x0014
subvendor=0x1912 subdevice=0x0014
vendor = 'Renesas Technology Corp.'
device = 'uPD720201 USB 3.0 Host Controller'
class = serial bus
subclass = USB

The usb controller (I passed thru to VM) is used with the card readers to test
smart cards.


Then I quit VM and destroy it:
sudo bhyvectl --destroy --vm=w10

After that I can't run it again:
sudo bhyve -c 8,sockets=2,cores=2,threads=2 -m 16G -H -w -S \
  -s 0,hostbridge \
  -s 3,ahci-cd,/neat/iso/Win10_21H1_Russian_x64.iso \
  -s 4,ahci-hd,/neat/bsd1/vms/w10.img \
  -s 5,virtio-net,tap0 \
  -s 7,passthru,216/0/0 \
  -s 29,fbuf,tcp=0.0.0.0:5900,w=1920,h=1200,wait \
  -s 30,xhci,tablet \
  -s 31,lpc \
  -l com1,stdio \
  -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
  w10

bhyve: Failed to map MSI-X table BAR on 216/0/0: Device busy
bhyve: failed to initialize BARs for PCI 216/0/0
device emulation initialization error: Device busy

On FreeBSD 13.0-RELEASE I have no trouble with restart VM.

My rc.conf, devfs.conf, loader.conf are the same as they were in FreeBSD
13.0-RELEASE:

[lanin@freebsd /etc]$ cat rc.conf 
...
kld_list="nvidia-modeset vmm nmdm"
devfs_system_ruleset="system"
cloned_interfaces="bridge0 tap0 tap1"
ifconfig_bridge0="addm igb0 addm tap0 addm tap1 up"

[lanin@freebsd /etc]$ cat devfs.conf 
[system=10]
add path 'usb/*' mode 0660 group operator

[lanin@freebsd /boot]$ cat loader.conf 
...
vmm_load="YES"
vmm_name="/boot/kernel/vmm.ko"
nmdm_load="YES"
if_bridge_load="YES"
if_tap_load="YES"
pptdevs="134/0/0 216/0/0/0"
...

I tried to wait...and restart devfs service - no effect.
I tried to unload/load vmm.ko. It has been unloaded succsessfully (it means
that bhyve's VM was destroyed well).
After vmm.ko has been loaded again, I tried to start VM, and got the same
error...

I tried to start VM without the device passed thru to it- in this case VM
started well.

I also tried the variant when I do not use passed thru device inside VM (I do
not connect anything to it inside the VM). Result is the same - VM is not
restarted.

I always disconnect all my readers from VM before I quit it.
When I quit vncviewer while VM is up, VM's window disappeared, but VM still
working...
And when I run vncviewer again, it connects to running VM. That's right.
When I quit VM while vncviewer is up, it quits automatically.

In fact, I have two passed thru devices, which differ only by PCI address:

[lanin@freebsd ~]$ sudo pciconf -lc |grep -A3 -e "ppt0" -e "ppt1"
ppt0@pci0:134:0:0:    class=0x0c0330 rev=0x03 hdr=0x00 vendor=0x1912
device=0x0014 subvendor=0x1912 subdevice=0x0014
    cap 01[50] = powerspec 3  supports D0 D3  current D0
    cap 05[70] = MSI supports 8 messages, 64 bit 
    cap 11[90] = MSI-X supports 8 messages
--
ppt1@pci0:216:0:0:    class=0x0c0330 rev=0x03 hdr=0x00 vendor=0x1912
device=0x0014 subvendor=0x1912 subdevice=0x0014
    cap 01[50] = powerspec 3  supports D0 D3  current D0
    cap 05[70] = MSI supports 8 messages, 64 bit 
    cap 11[90] = MSI-X supports 8 messages


One of them I passed thru to VM-1, and other - to VM-2.
I start these VMs simultaneously.
My first VM has: '-s 7,passthru,216/0/0'
and other: '-s 7,passthru,134/0/0'

I think, that message I have in the terminal is here:
/usr/src/usr.sbin/bhyve/pci_passthru.c, L:451:

-- 
You are receiving this mail because:
You are the assignee for the bug.