ERRATA CORRIGE "BUG !!!! X64 Exception Type - 06(#UD - Invalid Opcode) CPU Apic ID - 00000000 !!!! while trying to pass thru my devices with bhyve on FreeBSD 14-CURRENT"
Date: Thu, 10 Feb 2022 14:23:57 UTC
Hello.
I've bought an additional PCI to USB (Renesas) controller that I've
attached to the PCIe Gen3 x4 (Type 22110) of my MOBO Gigabyte Aorus Pro
with Intel I9 chipset and Nvidia Geforce RTX 2080 ti. On this controller
I've attached two USB disks and my KInect 2. The nVidia geforce RTX 2080 ti
is attached to the PCIe Gen3 x 4 (Type 2280). Check this image :
https://ibb.co/fC0jR1p
check by yourself the logs showing that my passed through devices are
working great :
dmesg --> https://pastebin.ubuntu.com/p/cSy7jkHdZz/
pciconf -vl --> https://pastebin.ubuntu.com/p/xZNggrz6D5/
On FreeBSD 14 the bhyve setup below works. I can boot Linux passing through
the devices that you see. But I have used a raw image file (not a physical
installation on a sata or nvme disk,because in this case it wont work and
for this I've opened another bug). With this I want to demonstrate that
there aren't any problems with my bios settings and with my passed thru
devices.
bhyve -S -c sockets=2,cores=2,threads=2 -m 8G -w -H \
-s 0,hostbridge \
-s 1,virtio-blk,/mnt/da1p2/bhyve/os/Linux/impish-cuda-11-5-nvidia-495.img \
-s 3:0,passthru,2/0/0 \
-s 3:1,passthru,2/0/1 \
-s 3:2,passthru,2/0/2 \
-s 3:3,passthru,2/0/3 \
-s 4:0,passthru,1/0/0 \
-s 9,virtio-net,tap0 \
-s 29,fbuf,tcp=0.0.0.0:5900,w=1440,h=900 \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
-l com1,stdio \
vm0
My host graphic card is not an Nvidia card. It's the graphic chipset
integrated on the motherboard (CoffeeLake-S GT2 [UHD Graphics 630); to use
it I've installed the xf86-video-intel driver with this xorg.conf file :
Section "Device"
Identifier "Card0"
Driver "intel"
BusID "PCI:0:2:0"
EndSection
so,now,let's to straight to this problem :
I've created a new VM with the goal to passthru my graphic card and my usb
controller from the host os (freebsd 14-CURRENT) to the guest os (freebsd
13R p7) and I've used the parameters below :
bhyve -S -c sockets=1,cores=2,threads=2 -m 4G -w -H -A \
-s 0,hostbridge \
-s 1,ahci-hd,/mnt/da2p2/bhyve/os/BSD/freebsd13vm.img \
-s 2,passthru,1/0/0 \
-s 3:0,passthru,2/0/0 \
-s 3:1,passthru,2/0/1 \
-s 3:2,passthru,2/0/2 \
-s 3:3,passthru,2/0/3 \
-s 8,virtio-net,tap1 \
-s 9,virtio-9p,sharename=/ \
-s 29,fbuf,tcp=0.0.0.0:5901,w=1440,h=900,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
-l com1,stdio \
vm1
inside /etc/rc.conf I've added the following parameters :
linux_enable="YES"
linux_mounts_enable="YES"
vm_enable="YES"
kdm5_enable="YES"
dbus_enable="YES"
sddm_enable="YES"
on /boot/loader.conf :
nvidia_load="YES"
vmm_load="YES"
nmdm_load="YES"
if_tap_load="YES"
if_bridge_load="YES"
bridgestp_load="YES"
linprocfs_load="YES"
linsysfs_load="YES"
tmpfs_load="YES"
verbose_loading="YES"
hw.vmm.topology.cores_per_package=8
kern.racct.enable=1
snd_hda_load="YES"
on /etc/sysctl.conf
net.link.tap.up_on_open=1
net.inet.ip.forwarding=1
net.inet.ip.random_id=1
kern.evdev.rcpt_mask=6
kern.init_shutdown_timeout="900"
kern.capmode_coredump=1
and with this xorg.conf file :
Section "ServerLayout"
Identifier "X.org Configured"
Screen 0 "Screen0" 0 0
#Screen 1 "Screen1" RightOf "Screen0"
InputDevice "Mouse0" "CorePointer"
InputDevice "Keyboard0" "CoreKeyboard"
EndSection
Section "Module"
Load "glx"
EndSection
Section "Files"
ModulePath "/usr/local/lib/xorg/modules"
FontPath "/usr/local/share/fonts/misc/"
FontPath "/usr/local/share/fonts/TTF/"
FontPath "/usr/local/share/fonts/OTF/"
FontPath "/usr/local/share/fonts/Type1/"
FontPath "/usr/local/share/fonts/100dpi/"
FontPath "/usr/local/share/fonts/75dpi/"
FontPath "catalogue:/usr/local/etc/X11/fontpath.d"
EndSection
Section "InputDevice"
Identifier "Mouse0"
Driver "mouse"
Option "Protocol" "auto"
Option "Device" "/dev/sysmouse"
Option "ZAxisMapping" "4 5 6 7"
EndSection
Section "Monitor"
Identifier "Monitor0"
VendorName "Monitor Vendor"
ModelName "Monitor Model"
EndSection
Section "Device"
### Available Driver options are:-
### Values: <i>: integer, <f>: float, <bool>: "True"/"False",
### <string>: "String", <freq>: "<f> Hz/kHz/MHz",
### <percent>: "<f>%"
### [arg]: arg optional
#Option "SWcursor" # [<bool>]
#Option "kmsdev" # <str>
#Option "ShadowFB" # [<bool>]
#Option "AccelMethod" # <str>
#Option "PageFlip" # [<bool>]
#Option "ZaphodHeads" # <str>
#Option "DoubleShadow" # [<bool>]
#Option "Atomic" # [<bool>]
Identifier "Card0"
Driver "nvidia"
BusID "PCI:0:4:0"
EndSection
this is what happens when I boot it :
Loading kernel...
/boot/kernel/kernel text=0x17b9e0 text=0xdd6d50 text=0x65ba0c data=0x140
data=0x1b9348+0x445cb8 syms=[0x8+0x178ed8+0x8+0x1990a2]
Loading configured modules...
snd_hda...if_bridge.../boot/kernel/if_bridge.ko size 0xfc48 at 0x2112000
loading required module 'bridgestp'
/boot/kernel/bridgestp.ko size 0x8178 at 0x2122000
tmpfs.../boot/kernel/tmpfs.ko size 0x14820 at 0x212b000
/etc/hostid.../etc/hostid size=0x25
/boot/entropy.../boot/entropy size=0x1000
if_tap...linprocfs.../boot/kernel/linprocfs.ko size 0x10e50 at 0x2141000
loading required module 'linux_common'
/boot/kernel/linux_common.ko size 0x2c650 at 0x2152000
nvidia.../boot/modules/nvidia.ko size 0x28c8640 at 0x217f000
loading required module 'linux'
/boot/kernel/linux.ko size 0x9e070 at 0x4a48000
nmdm.../boot/kernel/nmdm.ko size 0x4110 at 0x4ae7000
bridgestp...linsysfs.../boot/kernel/linsysfs.ko size 0x79b8 at 0x4aec000
vmm.../boot/kernel/vmm.ko size 0x5748f0 at 0x4af4000
Start @ 0xffffffff8037c000 ...
efi_check_space: Unable to expand staging area
efi_check_space: Unable to expand staging area
EFI framebuffer information:
addr, size 0xc3000000, 0x1000000
dimensions 1440 x 900
stride 1440
masks 0x00ff0000, 0x0000ff00, 0x000000ff, 0xff000000
!!!! X64 Exception Type - 06(#UD - Invalid Opcode) CPU Apic ID - 00000000
!!!!
RIP - 00000000000C000E, CS - 0000000000000038, RFLAGS - 0000000000010086
RAX - 0000000054A218D8, RCX - 0000000000000002, RDX - 0000000000000002
RBX - 0000000005069001, RSP - 00000000BFF58098, RBP - 00000000BFF58100
RSI - 0000000000001000, RDI - 0000000040001000
R8 - 0000000000000001, R9 - 00000000BFF580D8, R10 - 00000000B902D6E0
R11 - 0000000000000D80, R12 - 00000000BE819050, R13 - 0000000005069000
R14 - 00000000BE819058, R15 - 00000000BE819048
DS - 0000000000000030, ES - 0000000000000030, FS - 0000000000000030
GS - 0000000000000030, SS - 0000000000000030
CR0 - 0000000080010033, CR2 - 0000000000000000, CR3 - 00000000BFC01000
CR4 - 0000000000000668, CR8 - 0000000000000000
DR0 - 0000000000000000, DR1 - 0000000000000000, DR2 - 0000000000000000
DR3 - 0000000000000000, DR6 - 00000000FFFF0FF0, DR7 - 0000000000000400
GDTR - 00000000BFBEEA98 0000000000000047, LDTR - 0000000000000000
IDTR - 00000000BF832018 0000000000000FFF, TR - 0000000000000000
FXSAVE_STATE - 00000000BFF57CF0
!!!! Can't find image information. !!!!
it freezes here and I can't do anything anymore.
--
Mario.