bhyve PCI pass-through to Linux guest

[Sergey Manucharian]

Excerpts from Peter Grehan's message from Tue 22-Dec-15 13:09:
> Hi Sergey,
> 
> > This is pretty reproducible:
> >
> > I pass trough a PCI device (USB controller) to a Linux guest. It works
> > properly. Then I halt the VM, make sure that bhyve destroyed it and run
> > Windows guest with the same PCI device passed-through.
> >
> > Windows device manager does show the device, however, e.g. a flash drive
> > plugged in is not presented to Windows, instead it's being processed by
> > FreeBSD.
> >
> > After that it does not work in Linux guest as well. Kernel module (vmm)
> > unloading and reloading does not help.
> 
>   The flash drive being processed by FreeBSD would indicate that it has 
> ownership of the device. Would you be able to try a 'pciconf -vl' after 
> the Linux guest exists, and after the Windows guest exits ?

OK, I've experimented more and found that even with the same Linux VM
after halting it once, the pass-through device doesn't work when that
Linux guest is restarted again.

It looks that both host and guest report exactly the same data before and
after restarting the guest:

Linux - first run:
=================
$ lspci -v
00:07.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset
Family USB xHCI Host Controller (rev 04) (prog-if 30 [XHCI])
	Subsystem: Lenovo Device 21f3
	Flags: bus master, medium devsel, latency 0, IRQ 24
	Memory at c0010000 (64-bit, prefetchable) [size=64K]
	Capabilities: [70] Power Management version 2
	Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
	Kernel driver in use: xhci_hcd

$ lsusb 
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 0781:5530 SanDisk Corp. Cruzer
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

FreeBSD when Linux is running first time:
=========================================
$ pciconf -v
ppt0 at pci0:0:20:0:	class=0x0c0330 card=0x21f317aa chip=0x1e318086
rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '7 Series/C210 Series Chipset Family USB xHCI Host
Controller'
    class      = serial bus
    subclass   = USB

Linux - second run:
===================
$ lspci -v
00:07.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset
Family USB xHCI Host Controller (rev 04) (prog-if 30 [XHCI])
	Subsystem: Lenovo Device 21f3
	Flags: bus master, medium devsel, latency 0, IRQ 24
	Memory at c0010000 (64-bit, prefetchable) [size=64K]
	Capabilities: [70] Power Management version 2
	Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
	Kernel driver in use: xhci_hcd

FreeBSD when Linux is running second time:
==========================================
$ pciconf -lv
ppt0 at pci0:0:20:0:	class=0x0c0330 card=0x21f317aa chip=0x1e318086
rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '7 Series/C210 Series Chipset Family USB xHCI Host
Controller'
    class      = serial bus
    subclass   = USB


--
Thanks,
Sergey