bhyve pci passthrough usb controller

Sat Mar 24 02:30:10 UTC 2018

-- Start of PGP signed section.
> Hi, ..
> 
> I'm trying to pci passhthrough a usb controller to a bhyve windows 7
> virtualization.
> When I start the virtual machine it freezes the entire system. See
> console.log
> To be clear the system runs without any problems without the pci
> passthrough.
> 
> kernel configuration
> 
> include GENERIC
> ident?? OFL
> options IPFIREWALL
> options IPFIREWALL_VERBOSE
> options IPFIREWALL_DEFAULT_TO_ACCEPT
> options DUMMYNET
> options IPDIVERT
> options CONSPEED=115200
> 
> * note: removed xhci driver from GENERIC
> 
> 
> I've recompiled the kernel without the xhci driver. I've tested both usb
> controllers.
> Both "Sunrise Point-H USB 3.0 xHCI Controller" and "uPD720202 USB 3.0
> Host Controller", both showed
> the same error on the console screen.
> 
> # uname
> FreeBSD 11.1-STABLE #1 r331092M
> 
> # custom kernel
> include GENERIC
> ident?? OFL
> options IPFIREWALL
> options IPFIREWALL_VERBOSE
> options IPFIREWALL_DEFAULT_TO_ACCEPT
> options DUMMYNET
> options IPDIVERT
> options CONSPEED=115200
> 
> * note commented "device xhci" in generic
> 
> # svn info
> Path: .
> Working Copy Root Path: /usr/src
> URL: https://svn0.eu.freebsd.org/base/stable/11
> Relative URL: ^/stable/11
> Repository Root: https://svn0.eu.freebsd.org/base
> Repository UUID: ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
> Revision: 331092
> Node Kind: directory
> Schedule: normal
> Last Changed Author: eadler
> Last Changed Rev: 331092
> Last Changed Date: 2018-03-17 07:44:39 +0100 (Sat, 17 Mar 2018)
> 
> # vm passthru
> DEVICE???? BHYVE ID???? READY??????? DESCRIPTION
> hostb0???? 0/0/0??????? No?????????? Xeon E3-1200 v5/E3-1500 v5/6th Gen
> Core Processor Host Bridge/DRAM Registers
> pcib1????? 0/1/0??????? No?????????? Xeon E3-1200 v5/E3-1500 v5/6th Gen
> Core Processor PCIe Controller (x16)
> vgapci0??? 0/2/0??????? No?????????? HD Graphics 530
> ppt0?????? 0/20/0?????? Yes????????? Sunrise Point-H USB 3.0 xHCI Controller

0/20/0

> none0????? 0/20/2?????? No?????????? Sunrise Point-H Thermal subsystem

has a subdevice 0/20/2 that is not being passed, this may cause odd
things to happen.

> none1????? 0/22/0?????? No?????????? Sunrise Point-H CSME HECI
> ahci0????? 0/23/0?????? No?????????? Sunrise Point-H SATA controller
> [AHCI mode]
> pcib2????? 0/28/0?????? No?????????? Sunrise Point-H PCI Express Root Port
> pcib4????? 0/28/6?????? No?????????? Sunrise Point-H PCI Express Root Port
> isab0????? 0/31/0?????? No?????????? Sunrise Point-H LPC Controller
> none2????? 0/31/2?????? No?????????? Sunrise Point-H PMC
> hdac0????? 0/31/3?????? No?????????? Sunrise Point-H HD Audio
> none3????? 0/31/4?????? No?????????? Sunrise Point-H SMBus
> em0??????? 0/31/6?????? No?????????? Ethernet Connection (2) I219-V
> pcib3????? 2/0/0??????? No?????????? XIO2001 PCI Express-to-PCI Bridge
> xl0??????? 3/0/0??????? No?????????? 3c905B 100BaseTX [Cyclone]
> ppt1?????? 4/0/0??????? Yes????????? uPD720202 USB 3.0 Host Controller
> 
> ppt1 at pci0:4:0:0:??? class=0x0c0330 card=0x00151912 chip=0x00151912
> rev=0x02 hdr=0x00
> ??? vendor???? = 'Renesas Technology Corp.'/mailman
> ??? device???? = 'uPD720202 USB 3.0 Host Controller'
> ??? class????? = serial bus
> ??? subclass?? = USB
> 
> ppt0 at pci0:0:20:0:????? class=0x0c0330 card=0x06ba1028 chip=0xa12f8086
> rev=0x31 hdr=0x00
> ??? vendor???? = 'Intel Corporation'
> ??? device???? = 'Sunrise Point-H USB 3.0 xHCI Controller'
> ??? class????? = serial bus
> ??? subclass?? = USB
> 
> # grep -i pptdevfs loader.conf
> pptdevs="0/20/0 4/0/0"

add 0/20/2 to this

> 
> # vm configure
> passthru0="0/20/0"
> 
> # vm configure
> passthru0="4/0/0"
> 
> Any suggestions on what I'm doing wrong?
> 
> 
> * Sorry kinda new to this mailing list thing
> 
> 
> Op 23-03-18 om 18:25 schreef Julian Elischer:
> > On 23/3/18 11:40 pm, Wouter Snels wrote:
> >
> >
> > new wrapping is no better
> > It looks like the device you are passing though is not removed from
> > freebsd first.
> > but it's hard to tell.
> >
> >> |Hi, .. I'm trying to pci passhthrough a usb controller to a bhyve
> >> windows 7 virtualization. When I start the virtual machine it freezes
> >> the entire system and this shows up on the console. To be clear the
> >> system runs without any problems without the pci passthrough. |
> >> ||(ada0:ahcich0:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 e0 ff 40 40 29 00
> >> 00 00 00 00 (ada0:ahcich0:0:0:0): CAM status: CCB request was invalid
> >> (ada0:ahcich0:0:0:0): Error 22, Unretryable error
> >> (aprobe0:ahcich0:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00
> >> 00 00 00 (aprobe0:ahcich0:0:0:0): CAM status: CCB request was invalid
> >> (aprobe0:ahcich0:0:0:0): Error 22, Unretryable error ada0 at ahcich0 bus
> >> 0 scbus0 target 0 lun 0 ada0: <ST500DM002-1SB10A CC43> s/n ZA4292G6
> >> detached (aprobe0:ahcich0:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40
> >> 00 00 00 00 00 00 (aprobe0:ahcich0:0:0:0): CAM status: CCB request was
> >> invalid (aprobe0:ahcich0:0:0:0): Error 22, Unretryable error ahcich0:
> >> Poll timeout on slot 28 port 0 ahcich0: is 20000000 cs 10000000 ss
> >> 00000000 rs 10000000 tfd 1d0 serr 00000800 cmd 00005c17
> >> (aprobe0:ahcich0:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00
> >> 00 00 (aprobe0:ahcich0:0:0:0): CAM status: Command timeout
> >> (aprobe0:ahcich0:0:0:0): Error 5, Retries exhausted
> >> (ada0:ahcich0:0:0:0): SETFEATURES ENABLE RCACHE. ACB: ef aa 00 00 00 40
> >> 00 00 00 00 00 00 (ada0:ahcich0:0:0:0): CAM status: CCB request was
> >> invalid (ada0:ahcich0:0:0:0): Error 22, Unretryable error ahcich0: Poll
> >> timeout on slot 30 port 0 ahcich0: is 20000000 cs 40000000 ss 00000000
> >> rs 40000000 tfd 1d0 serr 00000800 cmd 00005e17 (aprobe0:ahcich0:0:0:0):
> >> SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
> >> (aprobe0:ahcich0:0:0:0): CAM status: Command timeout
> >> (aprobe0:ahcich0:0:0:0): Error 5, Retries exhausted
> >> (ada0:ahcich0:0:0:0): SETFEATURES ENABLE WCACHE. ACB: ef 02 00 00 00 40
> >> 00 00 00 00 00 00 (ada0:ahcich0:0:0:0): CAM status: CCB request was
> >> invalid (ada0:ahcich0:0:0:0): Error 22, Unretryable error
> >> (ada0:ahcich0:0:0:0): Periph destroyed em0: Watchdog timeout Queue[0]--
> >> resetting Interface is RUNNING and ACTIVE em0: TX Queue 0 ------ em0: hw
> >> tdh = 428, hw tdt = 471 em0: Tx Queue Status = -2147483648 em0: TX
> >> descriptors avail = 981 em0: Tx Descriptors avail failure = 0 em0: RX
> >> Queue 0 ------ em0: hw rdh = 912, hw rdt = 908 em0: RX discarded packets
> >> = 0 em0: RX Next to Check = 909 em0: RX Next to Refresh = 908 xl0:
> >> watchdog timeout em0: Watchdog timeout Queue[0]-- resetting Interface is
> >> RUNNING and ACTIVE em0: TX Queue 0 ------ em0: hw tdh = 0, hw tdt = 58
> >> em0: Tx Queue Status = -2147483648 em0: TX descriptors avail = 966 em0:
> >> Tx Descriptors avail failure = 0 em0: RX Queue 0 ------ em0: hw rdh = 0,
> >> hw rdt = 1023 em0: RX discarded packets = 0 em0: RX Next to Check = 0
> >> em0: RX Next to Refresh = 1023 em0: Watchdog timeout Queue[0]--
> >> resetting Interface is RUNNING and ACTIVE em0: TX Queue 0 ------ em0: hw
> >> tdh = 0, hw tdt = 58 em0: Tx Queue Status = -2147483648 em0: TX
> >> descriptors avail = 966 em0: Tx Descriptors avail failure = 0 em0: RX
> >> Queue 0 ------ em0: hw rdh = 0, hw rdt = 1023 em0: RX discarded packets
> >> = 0 em0: RX Next to Check = 0 em0: RX Next to Refresh = 1023 xl0:
> >> watchdog timeout em0: Watchdog timeout Queue[0]-- resetting Interface is
> >> RUNNING and ACTIVE em0: TX Queue 0 ------ em0: hw tdh = 0, hw tdt = 54
> >> em0: Tx Queue Status = -2147483648 em0: TX descriptors avail = 970 em0:
> >> Tx Descriptors avail failure = 0 em0: RX Queue 0 ------ em0: hw rdh = 0,
> >> hw rdt = 1023 em0: RX discarded packets = 0include GENERIC ident OFL
> >> options IPFIREWALL||||||
> >>
> >> |Hi, .. I'm trying to pci passhthrough a usb controller to a bhyve
> >> windows 7 virtualization. When I start the virtual machine it freezes
> >> the entire system and this shows up on the console. To be clear the
> >> system runs without any problems without the pci passthrough. |
> >> ||(ada0:ahcich0:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 e0 ff 40 40 29 00
> >> 00 00 00 00 (ada0:ahcich0:0:0:0): CAM status: CCB request was invalid
> >> (ada0:ahcich0:0:0:0): Error 22, Unretryable error
> >> (aprobe0:ahcich0:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00
> >> 00 00 00 (aprobe0:ahcich0:0:0:0): CAM status: CCB request was invalid
> >> (aprobe0:ahcich0:0:0:0): Error 22, Unretryable error ada0 at ahcich0 bus
> >> 0 scbus0 target 0 lun 0 ada0: <ST500DM002-1SB10A CC43> s/n ZA4292G6
> >> detached (aprobe0:ahcich0:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40
> >> 00 00 00 00 00 00 (aprobe0:ahcich0:0:0:0): CAM status: CCB request was
> >> invalid (aprobe0:ahcich0:0:0:0): Error 22, Unretryable error ahcich0:
> >> Poll timeout on slot 28 port 0 ahcich0: is 20000000 cs 10000000 ss
> >> 00000000 rs 10000000 tfd 1d0 serr 00000800 cmd 00005c17
> >> (aprobe0:ahcich0:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00
> >> 00 00 (aprobe0:ahcich0:0:0:0): CAM status: Command timeout
> >> (aprobe0:ahcich0:0:0:0): Error 5, Retries exhausted
> >> (ada0:ahcich0:0:0:0): SETFEATURES ENABLE RCACHE. ACB: ef aa 00 00 00 40
> >> 00 00 00 00 00 00 (ada0:ahcich0:0:0:0): CAM status: CCB request was
> >> invalid (ada0:ahcich0:0:0:0): Error 22, Unretryable error ahcich0: Poll
> >> timeout on slot 30 port 0 ahcich0: is 20000000 cs 40000000 ss 00000000
> >> rs 40000000 tfd 1d0 serr 00000800 cmd 00005e17 (aprobe0:ahcich0:0:0:0):
> >> SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
> >> (aprobe0:ahcich0:0:0:0): CAM status: Command timeout
> >> (aprobe0:ahcich0:0:0:0): Error 5, Retries exhausted
> >> (ada0:ahcich0:0:0:0): SETFEATURES ENABLE WCACHE. ACB: ef 02 00 00 00 40
> >> 00 00 00 00 00 00 (ada0:ahcich0:0:0:0): CAM status: CCB request was
> >> invalid (ada0:ahcich0:0:0:0): Error 22, Unretryable error
> >> (ada0:ahcich0:0:0:0): Periph destroyed em0: Watchdog timeout Queue[0]--
> >> resetting Interface is RUNNING and ACTIVE em0: TX Queue 0 ------ em0: hw
> >> tdh = 428, hw tdt = 471 em0: Tx Queue Status = -2147483648 em0: TX
> >> descriptors avail = 981 em0: Tx Descriptors avail failure = 0 em0: RX
> >> Queue 0 ------ em0: hw rdh = 912, hw rdt = 908 em0: RX discarded packets
> >> = 0 em0: RX Next to Check = 909 em0: RX Next to Refresh = 908 xl0:
> >> watchdog timeout em0: Watchdog timeout Queue[0]-- resetting Interface is
> >> RUNNING and ACTIVE em0: TX Queue 0 ------ em0: hw tdh = 0, hw tdt = 58
> >> em0: Tx Queue Status = -2147483648 em0: TX descriptors avail = 966 em0:
> >> Tx Descriptors avail failure = 0 em0: RX Queue 0 ------ em0: hw rdh = 0,
> >> hw rdt = 1023 em0: RX discarded packets = 0 em0: RX Next to Check = 0
> >> em0: RX Next to Refresh = 1023 em0: Watchdog timeout Queue[0]--
> >> resetting Interface is RUNNING and ACTIVE em0: TX Queue 0 ------ em0: hw
> >> tdh = 0, hw tdt = 58 em0: Tx Queue Status = -2147483648 em0: TX
> >> descriptors avail = 966 em0: Tx Descriptors avail failure = 0 em0: RX
> >> Queue 0 ------ em0: hw rdh = 0, hw rdt = 1023 em0: RX discarded packets
> >> = 0 em0: RX Next to Check = 0 em0: RX Next to Refresh = 1023 xl0:
> >> watchdog timeout em0: Watchdog timeout Queue[0]-- resetting Interface is
> >> RUNNING and ACTIVE em0: TX Queue 0 ------ em0: hw tdh = 0, hw tdt = 54
> >> em0: Tx Queue Status = -2147483648 em0: TX descriptors avail = 970 em0:
> >> Tx Descriptors avail failure = 0 em0: RX Queue 0 ------ em0: hw rdh = 0,
> >> hw rdt = 1023 em0: RX discarded packets = 0include GENERIC ident OFL
> >> options IPFIREWALL options IPFIREWALL_VERBOSE options
> >> IPFIREWALL_DEFAULT_TO_ACCEPT options DUMMYNET options IPDIVERT options
> >> CONSPEED=115200 em0: RX Next to Check = 0 em0: RX Next to Refresh =
> >> 1023 ||||
> >> |||I've recompiled the kernel without the xhci driver. I've tested both
> >> usb controllers. Both "|||||Sunrise Point-H USB 3.0 xHCI
> >> Controller|"||||| and "|||uPD720202 USB 3.0 Host Controller|", both
> >> showed the same error on the console screen.| # uname FreeBSD
> >> 11.1-STABLE #1 r331092M # custom kernel include GENERIC ident OFL
> >> options IPFIREWALL options IPFIREWALL_VERBOSE options
> >> IPFIREWALL_DEFAULT_TO_ACCEPT options DUMMYNET options IPDIVERT options
> >> CONSPEED=115200 * note commented "device xhci" in generic # svn info
> >> |||Path: . Working Copy Root Path: /usr/src URL:
> >> https://svn0.eu.freebsd.org/base/stable/11 Relative URL: ^/stable/11
> >> Repository Root: https://svn0.eu.freebsd.org/base Repository UUID:
> >> ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f Revision: 331092 Node Kind:
> >> directory Schedule: normal Last Changed Author: eadler Last Changed Rev:
> >> 331092 Last Changed Date: 2018-03-17 07:44:39 +0100 (Sat, 17 Mar 2018)
> >> |# vm passthru DEVICE BHYVE ID READY DESCRIPTION hostb0 0/0/0 No Xeon
> >> E3-1200 v5/E3-1500 v5/6th Gen Core Processor Host Bridge/DRAM Registers
> >> pcib1 0/1/0 No Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe
> >> Controller (x16) vgapci0 0/2/0 No HD Graphics 530 |||ppt0| 0/20/0 Yes
> >> Sunrise Point-H USB 3.0 xHCI Controller none0 0/20/2
> >> No Sunrise Point-H Thermal subsystem none1 0/22/0 No Sunrise Point-H
> >> CSME HECI ahci0 0/23/0 No Sunrise Point-H SATA controller [AHCI mode]
> >> pcib2 0/28/0 No Sunrise Point-H PCI Express Root Port pcib4 0/28/6 No
> >> Sunrise Point-H PCI Express Root Port isab0 0/31/0 No Sunrise Point-H
> >> LPC Controller none2 0/31/2 No Sunrise Point-H PMC hdac0 0/31/3 No
> >> Sunrise Point-H HD Audio none3 0/31/4 No Sunrise Point-H SMBus em0
> >> 0/31/6 No Ethernet Connection (2) I219-V pcib3 2/0/0 No XIO2001 PCI
> >> Express-to-PCI Bridge xl0 3/0/0 No 3c905B 100BaseTX [Cyclone] ppt1 4/0/0
> >> Yes uPD720202 USB 3.0 Host Controller ||ppt1 at pci0:4:0:0:
> >> class=0x0c0330 card=0x00151912 chip=0x00151912 rev=0x02
> >> hdr=0x00 vendor = 'Renesas Technology Corp.' device = 'uPD720202 USB 3.0
> >> Host Controller' class = serial bus subclass = USB
> >> |||ppt|0 at pci0:0:20:0: class=0x0c0330 card=0x06ba1028 chip=0xa12f8086
> >> rev=0x31 hdr=0x00 vendor = 'Intel Corporation' device = 'Sunrise Point-H
> >> USB 3.0 xHCI Controller' class = serial bus subclass = USB # grep -i
> >> pptdevfs loader.conf ||pptdevs="0/20/0 4/0/0" # vm configure
> >> ||passthru0="0/20/0"|
> >>
> >> |# vm configure ||passthru0="4/0/0"|
> >> ||Any suggestions on what I'm doing wrong?||
> >> ? Regards, Ofloo
> >> ||||||
> >>
> >> ||options IPFIREWALL_VERBOSE options IPFIREWALL_DEFAULT_TO_ACCEPT
> >> options
> >> DUMMYNET options IPDIVERT options CONSPEED=115200 em0: RX Next to Check
> >> = 0 em0: RX Next to Refresh = 1023 ||||
> >> |||I've recompiled the kernel without the xhci driver. I've tested both
> >> usb controllers. Both "|||||Sunrise Point-H USB 3.0 xHCI
> >> Controller|"||||| and "|||uPD720202 USB 3.0 Host Controller|", both
> >> showed the same error on the console screen.| # uname FreeBSD
> >> 11.1-STABLE #1 r331092M # custom kernel include GENERIC ident OFL
> >> options IPFIREWALL options IPFIREWALL_VERBOSE options
> >> IPFIREWALL_DEFAULT_TO_ACCEPT options DUMMYNET options IPDIVERT options
> >> CONSPEED=115200 * note commented "device xhci" in generic # svn info
> >> |||Path: . Working Copy Root Path: /usr/src URL:
> >> https://svn0.eu.freebsd.org/base/stable/11 Relative URL: ^/stable/11
> >> Repository Root: https://svn0.eu.freebsd.org/base Repository UUID:
> >> ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f Revision: 331092 Node Kind:
> >> directory Schedule: normal Last Changed Author: eadler Last Changed Rev:
> >> 331092 Last Changed Date: 2018-03-17 07:44:39 +0100 (Sat, 17 Mar 2018)
> >> |# vm passthru DEVICE BHYVE ID READY DESCRIPTION hostb0 0/0/0 No Xeon
> >> E3-1200 v5/E3-1500 v5/6th Gen Core Processor Host Bridge/DRAM Registers
> >> pcib1 0/1/0 No Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe
> >> Controller (x16) vgapci0 0/2/0 No HD Graphics 530 |||ppt0| 0/20/0 Yes
> >> Sunrise Point-H USB 3.0 xHCI Controller none0 0/20/2
> >> No Sunrise Point-H Thermal subsystem none1 0/22/0 No Sunrise Point-H
> >> CSME HECI ahci0 0/23/0 No Sunrise Point-H SATA controller [AHCI mode]
> >> pcib2 0/28/0 No Sunrise Point-H PCI Express Root Port pcib4 0/28/6 No
> >> Sunrise Point-H PCI Express Root Port isab0 0/31/0 No Sunrise Point-H
> >> LPC Controller none2 0/31/2 No Sunrise Point-H PMC hdac0 0/31/3 No
> >> Sunrise Point-H HD Audio none3 0/31/4 No Sunrise Point-H SMBus em0
> >> 0/31/6 No Ethernet Connection (2) I219-V pcib3 2/0/0 No XIO2001 PCI
> >> Express-to-PCI Bridge xl0 3/0/0 No 3c905B 100BaseTX [Cyclone] ppt1 4/0/0
> >> Yes uPD720202 USB 3.0 Host Controller ||ppt1 at pci0:4:0:0:
> >> class=0x0c0330 card=0x00151912 chip=0x00151912 rev=0x02
> >> hdr=0x00 vendor = 'Renesas Technology Corp.' device = 'uPD720202 USB 3.0
> >> Host Controller' class = serial bus subclass = USB
> >> |||ppt|0 at pci0:0:20:0: class=0x0c0330 card=0x06ba1028 chip=0xa12f8086
> >> rev=0x31 hdr=0x00 vendor = 'Intel Corporation' device = 'Sunrise Point-H
> >> USB 3.0 xHCI Controller' class = serial bus subclass = USB # grep -i
> >> pptdevfs loader.conf ||pptdevs="0/20/0 4/0/0" # vm configure
> >> ||passthru0="0/20/0"|
> >>
> >> |# vm configure ||passthru0="4/0/0"|
> >> ||Any suggestions on what I'm doing wrong?||
> >>
> >> edit: previous wrapping was all wrong.
> >> ?? Regards, Ofloo
> >> ||||||
> >>
> >>
> 
-- End of PGP section, PGP failed!

-- 
Rod Grimes                                                 rgrimes at freebsd.org