GPU Passthrough with FreeBSD 14.2 bhyve and NVidia Quadro RTX 6000/8000

From: Shamim Shahriar <shamim.shahriar_at_gmail.com>
Date: Mon, 24 Mar 2025 14:21:48 UTC
Good afternoon everyone.

I am trying to have VMs with GPU passthrough. The setup is a Dell server
with NVidia Quadro RTX 6000/8000 installed already. I have checked the
device IDs and put pptdevs in place

# cat /boot/loader.conf
pptdevs="18/0/0 19/0/0"
pptdevs2="72/0/0 73/0/0"

this is showing the GPUs as pptdev on the pciconf

ppt0@pci0:18:0:0:       class=0x030200 rev=0xa1 hdr=0x00 vendor=0x10de
device=0x1e78 subvendor=0x10de subdevice=0x13d8
    vendor     = 'NVIDIA Corporation'
    device     = 'TU102GL [Quadro RTX 6000/8000]'
    class      = display
    subclass   = 3D
ppt1@pci0:19:0:0:       class=0x030200 rev=0xa1 hdr=0x00 vendor=0x10de
device=0x1e78 subvendor=0x10de subdevice=0x13d8
    vendor     = 'NVIDIA Corporation'
    device     = 'TU102GL [Quadro RTX 6000/8000]'
    class      = display
    subclass   = 3D

as I am using vm-bhyve, I have put the configuration as below

# cat /mnt/VMs/jagadish/jagadish.conf
loader="uefi"
cpu=16
memory=128G
xhci_mouse="yes"
debug="true"

graphics="yes"
graphics_listen="127.0.0.1"
graphics_port="5920"
graphics_res="1024x768"
graphics_wait="no"
#graphics_vga="io"

network0_type="virtio-net"
network0_switch="swUNI"
network0_mac="58:9c:fc:06:3f:af"

disk0_type="nvme"
disk0_name="jagadish-disk0.img"
#disk0_size="128G"

uuid="966e909b-1293-11ef-a9a4-e4434bfe34de"

passthru0="19/0/0=6:0"

bhyve_options="-A -H -P"
#END

however, when I start the vm (for OS installation to start with), it shows
it is running but I am unable to access the VNC for a while, then when I
finally manage to connect via VNC, there is nothing on the screen, just
blank dark screen

the vm-bhyve.log shows

Mar 24 13:59:26: initialising
Mar 24 13:59:26:  [loader: uefi]
Mar 24 13:59:26:  [cpu: 16]
Mar 24 13:59:26:  [memory: 128G]
Mar 24 13:59:26:  [hostbridge: standard]
Mar 24 13:59:26:  [com ports: com1]
Mar 24 13:59:26:  [uuid: 966e909b-1293-11ef-a9a4-e4434bfe34de]
Mar 24 13:59:26:  [debug mode: true]
Mar 24 13:59:26:  [primary disk: jagadish-disk0.img]
Mar 24 13:59:26:  [primary disk dev: file]
Mar 24 13:59:26: initialising network device tap0
Mar 24 13:59:26: adding tap0 -> vm-swUNI (swUNI addm)
Mar 24 13:59:26: bring up tap0 -> vm-swUNI (swUNI addm)
Mar 24 13:59:26: booting
Mar 24 13:59:26:  [bhyve options: -c 16 -m 128G -AHPw -l
bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -A -H -P -U
966e909b-1293-11ef-a9a4-e4434bfe34de -u -S]
Mar 24 13:59:26:  [bhyve devices: -s 0,hostbridge -s 31,lpc -s
4:0,nvme,/mnt/VMs/jagadish/jagadish-disk0.img -s
5:0,virtio-net,tap0,mac=58:9c:fc:06:3f:af -s 6:0,passthru,19/0/0 -s
7:0,fbuf,tcp=127.0.0.1:5920,w=1024,h=768 -s 8:0,xhci,tablet]
Mar 24 13:59:26:  [bhyve console: -l com1,/dev/nmdm-jagadish.1A]
Mar 24 13:59:26:  [bhyve iso device: -s
3:0,ahci-cd,/mnt/VMs/.iso/lubuntu-24.04.1-desktop-amd64.iso,ro]
Mar 24 13:59:26: starting bhyve (run 1)

based on what I can see, and a little out of desparation, I decided to run
the installer in the foreground, and below is what I got

# vm install -f jagadish FreeBSD-14.2-RELEASE-amd64-disc1.iso
Starting jagadish
  * found guest in /mnt/VMs/jagadish
  * booting...
fbuf frame buffer base: 0x112245400000 [sz 16777216]

it stays there for as long as I wait until I poweroff the vm.

tried with Debian installer,
# vm install -f jagadish debian-12.5.0-amd64-netinst.iso
Starting jagadish
  * found guest in /mnt/VMs/jagadish
  * booting...
fbuf frame buffer base: 0x2747e2400000 [sz 16777216]


ideally I need to install Debian for my user base on these GPU based
systems. But even that is proving to be impossible since the installer does
not move any further from where it is.

Any thoughts/ideas/suggestions what I else I can try to make this work?
Anything that you have tried that worked? have I missed something?

Would appreciate any and all thoughts/suggestions

best regards
SS