Re: GPU Passthrough with FreeBSD 14.2 bhyve and NVidia Quadro RTX 6000/8000

Hi Mario

I will try the options you have used later and let you know how it goes.
Not sure if this is intentional or typo, but I don't recall any bhyve-win
binary in my system.

Best regards
SS

On Mon, 24 Mar 2025 at 14:59, Mario Marietto <marietto2008@gmail.com> wrote:

> Oh sorry. I forgot to add the following bhyve parameters necessary to
> passthru my GPU :D
>
> -s 8:0,passthru,2/0/0 \
> -s 8:1,passthru,2/0/1 \
> -s 8:2,passthru,2/0/2 \
> -s 8:3,passthru,2/0/3 \
>
> On Mon, Mar 24, 2025 at 3:57 PM Mario Marietto <marietto2008@gmail.com>
> wrote:
>
>> Usually I use this kind of script to launch a bhyve vm :
>>
>> #!/bin/sh
>>
>> setxkbmap it
>> kldload vmm.ko
>> vms="$(ls /dev/vmm/*)"
>> vncs="$(ps ax | awk '/vncviewer [0]/{print $6}')"
>> kldload vmm.ko
>>
>> if ! pciconf -l pci0:2:0:0 | grep -q "^ppt"; then
>> echo "rtx 2080ti slot 2/0/0 is not attached to ppt,attaching..."
>> kldload nvidia-modeset
>> devctl clear driver -f pci0:2:0:0
>> devctl set driver -f pci0:2:0:0 ppt
>> else
>> echo "rtx 2080ti slot 2/0/0 is already attached to ppt"
>> fi
>>
>> if ! pciconf -l pci0:2:0:1 | grep -q "^ppt"; then
>> echo "rtx 2080ti slot 2/0/1 is not attached to ppt,attaching..."
>> devctl clear driver -f pci0:2:0:1
>> devctl set driver -f pci0:2:0:1 ppt
>> else
>> echo "rtx 2080ti slot 2/0/1 is already attached to ppt"
>> fi
>>
>> if ! pciconf -l pci0:2:0:2 | grep -q "^ppt"; then
>> echo "rtx 2080ti slot 2/0/2 is not attached to ppt,attaching..."
>> devctl clear driver -f pci0:2:0:2
>> devctl set driver -f pci0:2:0:2 ppt
>> else
>> echo "rtx 2080ti slot 2/0/2 is already attached to ppt"
>> fi
>>
>> if ! pciconf -l pci0:2:0:3 | grep -q "^ppt"; then
>> echo "rtx 2080ti slot 2/0/3 is not attached to ppt,attaching..."
>> devctl clear driver -f pci0:2:0:3
>> devctl set driver -f pci0:2:0:3 ppt
>> else
>> echo "rtx 2080ti slot 2/0/3 is already attached to ppt"
>> fi
>>
>> echo "rtx 2080ti is fully attached to ppt"
>>
>> for vm in $vms; do
>>                 session="${vm##*/}"
>>                 echo "bhyve session = $session"
>>                 echo "vnc session = $vncs"
>>                 if ! printf '%s\n' "${vncs}" | grep "${session#vm}"; then
>>
>>                 printf 'VNC session not found,destroying ghost vms\n'
>>                                 bhyvectl --vm=$session --destroy
>>                 else
>>                                 printf 'Found VNC session %s\n'
>> "${session},no ghost vms found,not destroying them"
>>                 fi
>> done
>>
>> vmdisk0=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>> (AM13N4CZ)/ && d{print d}'`
>> echo "Seagate M3 Portable 1.8T ; $vmdisk0"
>>
>> vmdisk1=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>> (2015040204055E)/ && d{print d}'`
>> echo "TOSHIBA External USB 3.0 1.8T ; $vmdisk1"
>>
>> vmdisk2=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>> (2027285F1175)/ && d{print d}'`
>> echo "CT1000P1SSD8 ; $vmdisk2"
>>
>> vmdisk3=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>> (20130508005976F)/ && d{print d}'`
>> echo "TOSHIBA External USB 3.0 932 GB ; $vmdisk3"
>>
>> vmdisk4=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>> (BE0191510218)/ && d{print d}'`
>> echo "G-DRIVE USB ; $vmdisk4"
>>
>> vmdisk5=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>> (38234B4237354B45)/ && d{print d}'`
>> echo "Elements 25A3 ; $vmdisk5"
>>
>> vmdisk6=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>> (WD-WCAV2X797309)/ && d{print d}'`
>> echo "WDC WD3200AAJS-00L7A0 ; 298 GB ; $vmdisk6"
>>
>> vmdisk7=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>> (20140108006C)/ && d{print d}'`
>> echo "Corsair Force 3 SSD ; $vmdisk7"
>>
>> vmdisk8=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>> (1924E50B2AE5)/ && d{print d}'`
>> echo "CT500MX500SSD4 ; $vmdisk8"
>>
>> vmdisk9=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>> (0774921DDC4200A6)/ && d{print d}'`
>> echo "SanDisk Cruzer-15GB ; $vmdisk9"
>>
>> vmdisk10=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>> (60A44D4138D8F311190A0149)/ && d{print d}'`
>> echo "Kingston DataTraveler 2.0 ; $vmdisk10"
>>
>> vmdisk11=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>> (575845483038524844323238)/ && d{print d}'`
>> echo "WD 2500BMV External ; $vmdisk11"
>>
>> vmdisk12=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>> (57442E575845323039544433303334)/ && d{print d}'`
>> echo "WD 3200BMV External ; $vmdisk12"
>>
>> vmdisk13=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>> (2414E989076B)/ && d{print d}'`
>> echo "CT500BX500SSD1 ; $vmdisk13"
>>
>> /usr/sbin/./bhyve-win -S -c sockets=4,cores=1,threads=1 -m 4G -w -H -A \
>> -s 0,hostbridge \
>> -s 1,ahci-hd,/dev/$vmdisk13 \
>> -s 10,virtio-net,tap6 \
>> -s 11,virtio-9p,sharename=/ \
>> -s 29,fbuf,tcp=0.0.0.0:5906,w=1600,h=950 \
>> -s 31,lpc \
>> -l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \
>> vm0:6 < /dev/null & sleep 5 && vncviewer 0:6 &
>>
>> As you can see,each vm has a unique ID (connected with the vncviewer
>> instance) and the vncviewer window is launched as soon as the bhyve vm is
>> called...and my Geforce RTX 2080 ti is passed through inside the vm.
>>
>>
>> On Mon, Mar 24, 2025 at 3:47 PM Mario Marietto <marietto2008@gmail.com>
>> wrote:
>>
>>> First of all,you could use a more flexible technique than using the
>>> pptdevs on /boot/loader.conf. I would use a script like this :
>>>
>>> 1)
>>>
>>> if ! pciconf -l pci0:18:0:0 | grep -q "^ppt"; then
>>> echo "18/0/0 is not attached to ppt,attaching..."
>>> devctl clear driver -f pci0:18:0:0
>>> devctl set driver -f pci0:18:0:0 ppt
>>> else
>>> echo "18/0/0 already attached to ppt"
>>> fi
>>>
>>> 2) I would not like to use vm-bhyve. It adds only some confusion...
>>>
>>> On Mon, Mar 24, 2025 at 3:22 PM Shamim Shahriar <
>>> shamim.shahriar@gmail.com> wrote:
>>>
>>>> Good afternoon everyone.
>>>>
>>>> I am trying to have VMs with GPU passthrough. The setup is a Dell
>>>> server with NVidia Quadro RTX 6000/8000 installed already. I have checked
>>>> the device IDs and put pptdevs in place
>>>>
>>>> # cat /boot/loader.conf
>>>> pptdevs="18/0/0 19/0/0"
>>>> pptdevs2="72/0/0 73/0/0"
>>>>
>>>> this is showing the GPUs as pptdev on the pciconf
>>>>
>>>> ppt0@pci0:18:0:0:       class=0x030200 rev=0xa1 hdr=0x00 vendor=0x10de
>>>> device=0x1e78 subvendor=0x10de subdevice=0x13d8
>>>>     vendor     = 'NVIDIA Corporation'
>>>>     device     = 'TU102GL [Quadro RTX 6000/8000]'
>>>>     class      = display
>>>>     subclass   = 3D
>>>> ppt1@pci0:19:0:0:       class=0x030200 rev=0xa1 hdr=0x00 vendor=0x10de
>>>> device=0x1e78 subvendor=0x10de subdevice=0x13d8
>>>>     vendor     = 'NVIDIA Corporation'
>>>>     device     = 'TU102GL [Quadro RTX 6000/8000]'
>>>>     class      = display
>>>>     subclass   = 3D
>>>>
>>>> as I am using vm-bhyve, I have put the configuration as below
>>>>
>>>> # cat /mnt/VMs/jagadish/jagadish.conf
>>>> loader="uefi"
>>>> cpu=16
>>>> memory=128G
>>>> xhci_mouse="yes"
>>>> debug="true"
>>>>
>>>> graphics="yes"
>>>> graphics_listen="127.0.0.1"
>>>> graphics_port="5920"
>>>> graphics_res="1024x768"
>>>> graphics_wait="no"
>>>> #graphics_vga="io"
>>>>
>>>> network0_type="virtio-net"
>>>> network0_switch="swUNI"
>>>> network0_mac="58:9c:fc:06:3f:af"
>>>>
>>>> disk0_type="nvme"
>>>> disk0_name="jagadish-disk0.img"
>>>> #disk0_size="128G"
>>>>
>>>> uuid="966e909b-1293-11ef-a9a4-e4434bfe34de"
>>>>
>>>> passthru0="19/0/0=6:0"
>>>>
>>>> bhyve_options="-A -H -P"
>>>> #END
>>>>
>>>> however, when I start the vm (for OS installation to start with), it
>>>> shows it is running but I am unable to access the VNC for a while, then
>>>> when I finally manage to connect via VNC, there is nothing on the screen,
>>>> just blank dark screen
>>>>
>>>> the vm-bhyve.log shows
>>>>
>>>> Mar 24 13:59:26: initialising
>>>> Mar 24 13:59:26:  [loader: uefi]
>>>> Mar 24 13:59:26:  [cpu: 16]
>>>> Mar 24 13:59:26:  [memory: 128G]
>>>> Mar 24 13:59:26:  [hostbridge: standard]
>>>> Mar 24 13:59:26:  [com ports: com1]
>>>> Mar 24 13:59:26:  [uuid: 966e909b-1293-11ef-a9a4-e4434bfe34de]
>>>> Mar 24 13:59:26:  [debug mode: true]
>>>> Mar 24 13:59:26:  [primary disk: jagadish-disk0.img]
>>>> Mar 24 13:59:26:  [primary disk dev: file]
>>>> Mar 24 13:59:26: initialising network device tap0
>>>> Mar 24 13:59:26: adding tap0 -> vm-swUNI (swUNI addm)
>>>> Mar 24 13:59:26: bring up tap0 -> vm-swUNI (swUNI addm)
>>>> Mar 24 13:59:26: booting
>>>> Mar 24 13:59:26:  [bhyve options: -c 16 -m 128G -AHPw -l
>>>> bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -A -H -P -U
>>>> 966e909b-1293-11ef-a9a4-e4434bfe34de -u -S]
>>>> Mar 24 13:59:26:  [bhyve devices: -s 0,hostbridge -s 31,lpc -s
>>>> 4:0,nvme,/mnt/VMs/jagadish/jagadish-disk0.img -s
>>>> 5:0,virtio-net,tap0,mac=58:9c:fc:06:3f:af -s 6:0,passthru,19/0/0 -s
>>>> 7:0,fbuf,tcp=127.0.0.1:5920,w=1024,h=768 -s 8:0,xhci,tablet]
>>>> Mar 24 13:59:26:  [bhyve console: -l com1,/dev/nmdm-jagadish.1A]
>>>> Mar 24 13:59:26:  [bhyve iso device: -s
>>>> 3:0,ahci-cd,/mnt/VMs/.iso/lubuntu-24.04.1-desktop-amd64.iso,ro]
>>>> Mar 24 13:59:26: starting bhyve (run 1)
>>>>
>>>> based on what I can see, and a little out of desparation, I decided to
>>>> run the installer in the foreground, and below is what I got
>>>>
>>>> # vm install -f jagadish FreeBSD-14.2-RELEASE-amd64-disc1.iso
>>>> Starting jagadish
>>>>   * found guest in /mnt/VMs/jagadish
>>>>   * booting...
>>>> fbuf frame buffer base: 0x112245400000 [sz 16777216]
>>>>
>>>> it stays there for as long as I wait until I poweroff the vm.
>>>>
>>>> tried with Debian installer,
>>>> # vm install -f jagadish debian-12.5.0-amd64-netinst.iso
>>>> Starting jagadish
>>>>   * found guest in /mnt/VMs/jagadish
>>>>   * booting...
>>>> fbuf frame buffer base: 0x2747e2400000 [sz 16777216]
>>>>
>>>>
>>>> ideally I need to install Debian for my user base on these GPU based
>>>> systems. But even that is proving to be impossible since the installer does
>>>> not move any further from where it is.
>>>>
>>>> Any thoughts/ideas/suggestions what I else I can try to make this work?
>>>> Anything that you have tried that worked? have I missed something?
>>>>
>>>> Would appreciate any and all thoughts/suggestions
>>>>
>>>> best regards
>>>> SS
>>>>
>>>
>>>
>>> --
>>> Mario.
>>>
>>
>>
>> --
>> Mario.
>>
>
>
> --
> Mario.
>