Re: GPU Passthrough with FreeBSD 14.2 bhyve and NVidia Quadro RTX 6000/8000
- In reply to: Shamim Shahriar : "Re: GPU Passthrough with FreeBSD 14.2 bhyve and NVidia Quadro RTX 6000/8000"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Mon, 24 Mar 2025 17:34:58 UTC
Hi Mario, tried options similar to what you suggested, no difference to the
outcome. Stuck where it was before.
Best regards
SS
On Mon, 24 Mar 2025 at 15:29, Shamim Shahriar <shamim.shahriar@gmail.com>
wrote:
> Hi Mario
>
> I will try the options you have used later and let you know how it goes.
> Not sure if this is intentional or typo, but I don't recall any bhyve-win
> binary in my system.
>
> Best regards
> SS
>
> On Mon, 24 Mar 2025 at 14:59, Mario Marietto <marietto2008@gmail.com>
> wrote:
>
>> Oh sorry. I forgot to add the following bhyve parameters necessary to
>> passthru my GPU :D
>>
>> -s 8:0,passthru,2/0/0 \
>> -s 8:1,passthru,2/0/1 \
>> -s 8:2,passthru,2/0/2 \
>> -s 8:3,passthru,2/0/3 \
>>
>> On Mon, Mar 24, 2025 at 3:57 PM Mario Marietto <marietto2008@gmail.com>
>> wrote:
>>
>>> Usually I use this kind of script to launch a bhyve vm :
>>>
>>> #!/bin/sh
>>>
>>> setxkbmap it
>>> kldload vmm.ko
>>> vms="$(ls /dev/vmm/*)"
>>> vncs="$(ps ax | awk '/vncviewer [0]/{print $6}')"
>>> kldload vmm.ko
>>>
>>> if ! pciconf -l pci0:2:0:0 | grep -q "^ppt"; then
>>> echo "rtx 2080ti slot 2/0/0 is not attached to ppt,attaching..."
>>> kldload nvidia-modeset
>>> devctl clear driver -f pci0:2:0:0
>>> devctl set driver -f pci0:2:0:0 ppt
>>> else
>>> echo "rtx 2080ti slot 2/0/0 is already attached to ppt"
>>> fi
>>>
>>> if ! pciconf -l pci0:2:0:1 | grep -q "^ppt"; then
>>> echo "rtx 2080ti slot 2/0/1 is not attached to ppt,attaching..."
>>> devctl clear driver -f pci0:2:0:1
>>> devctl set driver -f pci0:2:0:1 ppt
>>> else
>>> echo "rtx 2080ti slot 2/0/1 is already attached to ppt"
>>> fi
>>>
>>> if ! pciconf -l pci0:2:0:2 | grep -q "^ppt"; then
>>> echo "rtx 2080ti slot 2/0/2 is not attached to ppt,attaching..."
>>> devctl clear driver -f pci0:2:0:2
>>> devctl set driver -f pci0:2:0:2 ppt
>>> else
>>> echo "rtx 2080ti slot 2/0/2 is already attached to ppt"
>>> fi
>>>
>>> if ! pciconf -l pci0:2:0:3 | grep -q "^ppt"; then
>>> echo "rtx 2080ti slot 2/0/3 is not attached to ppt,attaching..."
>>> devctl clear driver -f pci0:2:0:3
>>> devctl set driver -f pci0:2:0:3 ppt
>>> else
>>> echo "rtx 2080ti slot 2/0/3 is already attached to ppt"
>>> fi
>>>
>>> echo "rtx 2080ti is fully attached to ppt"
>>>
>>> for vm in $vms; do
>>> session="${vm##*/}"
>>> echo "bhyve session = $session"
>>> echo "vnc session = $vncs"
>>> if ! printf '%s\n' "${vncs}" | grep "${session#vm}";
>>> then
>>> printf 'VNC session not found,destroying ghost vms\n'
>>> bhyvectl --vm=$session --destroy
>>> else
>>> printf 'Found VNC session %s\n'
>>> "${session},no ghost vms found,not destroying them"
>>> fi
>>> done
>>>
>>> vmdisk0=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>>> (AM13N4CZ)/ && d{print d}'`
>>> echo "Seagate M3 Portable 1.8T ; $vmdisk0"
>>>
>>> vmdisk1=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>>> (2015040204055E)/ && d{print d}'`
>>> echo "TOSHIBA External USB 3.0 1.8T ; $vmdisk1"
>>>
>>> vmdisk2=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>>> (2027285F1175)/ && d{print d}'`
>>> echo "CT1000P1SSD8 ; $vmdisk2"
>>>
>>> vmdisk3=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>>> (20130508005976F)/ && d{print d}'`
>>> echo "TOSHIBA External USB 3.0 932 GB ; $vmdisk3"
>>>
>>> vmdisk4=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>>> (BE0191510218)/ && d{print d}'`
>>> echo "G-DRIVE USB ; $vmdisk4"
>>>
>>> vmdisk5=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>>> (38234B4237354B45)/ && d{print d}'`
>>> echo "Elements 25A3 ; $vmdisk5"
>>>
>>> vmdisk6=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>>> (WD-WCAV2X797309)/ && d{print d}'`
>>> echo "WDC WD3200AAJS-00L7A0 ; 298 GB ; $vmdisk6"
>>>
>>> vmdisk7=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>>> (20140108006C)/ && d{print d}'`
>>> echo "Corsair Force 3 SSD ; $vmdisk7"
>>>
>>> vmdisk8=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>>> (1924E50B2AE5)/ && d{print d}'`
>>> echo "CT500MX500SSD4 ; $vmdisk8"
>>>
>>> vmdisk9=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>>> (0774921DDC4200A6)/ && d{print d}'`
>>> echo "SanDisk Cruzer-15GB ; $vmdisk9"
>>>
>>> vmdisk10=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>>> (60A44D4138D8F311190A0149)/ && d{print d}'`
>>> echo "Kingston DataTraveler 2.0 ; $vmdisk10"
>>>
>>> vmdisk11=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>>> (575845483038524844323238)/ && d{print d}'`
>>> echo "WD 2500BMV External ; $vmdisk11"
>>>
>>> vmdisk12=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>>> (57442E575845323039544433303334)/ && d{print d}'`
>>> echo "WD 3200BMV External ; $vmdisk12"
>>>
>>> vmdisk13=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>>> (2414E989076B)/ && d{print d}'`
>>> echo "CT500BX500SSD1 ; $vmdisk13"
>>>
>>> /usr/sbin/./bhyve-win -S -c sockets=4,cores=1,threads=1 -m 4G -w -H -A \
>>> -s 0,hostbridge \
>>> -s 1,ahci-hd,/dev/$vmdisk13 \
>>> -s 10,virtio-net,tap6 \
>>> -s 11,virtio-9p,sharename=/ \
>>> -s 29,fbuf,tcp=0.0.0.0:5906,w=1600,h=950 \
>>> -s 31,lpc \
>>> -l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \
>>> vm0:6 < /dev/null & sleep 5 && vncviewer 0:6 &
>>>
>>> As you can see,each vm has a unique ID (connected with the vncviewer
>>> instance) and the vncviewer window is launched as soon as the bhyve vm is
>>> called...and my Geforce RTX 2080 ti is passed through inside the vm.
>>>
>>>
>>> On Mon, Mar 24, 2025 at 3:47 PM Mario Marietto <marietto2008@gmail.com>
>>> wrote:
>>>
>>>> First of all,you could use a more flexible technique than using the
>>>> pptdevs on /boot/loader.conf. I would use a script like this :
>>>>
>>>> 1)
>>>>
>>>> if ! pciconf -l pci0:18:0:0 | grep -q "^ppt"; then
>>>> echo "18/0/0 is not attached to ppt,attaching..."
>>>> devctl clear driver -f pci0:18:0:0
>>>> devctl set driver -f pci0:18:0:0 ppt
>>>> else
>>>> echo "18/0/0 already attached to ppt"
>>>> fi
>>>>
>>>> 2) I would not like to use vm-bhyve. It adds only some confusion...
>>>>
>>>> On Mon, Mar 24, 2025 at 3:22 PM Shamim Shahriar <
>>>> shamim.shahriar@gmail.com> wrote:
>>>>
>>>>> Good afternoon everyone.
>>>>>
>>>>> I am trying to have VMs with GPU passthrough. The setup is a Dell
>>>>> server with NVidia Quadro RTX 6000/8000 installed already. I have checked
>>>>> the device IDs and put pptdevs in place
>>>>>
>>>>> # cat /boot/loader.conf
>>>>> pptdevs="18/0/0 19/0/0"
>>>>> pptdevs2="72/0/0 73/0/0"
>>>>>
>>>>> this is showing the GPUs as pptdev on the pciconf
>>>>>
>>>>> ppt0@pci0:18:0:0: class=0x030200 rev=0xa1 hdr=0x00
>>>>> vendor=0x10de device=0x1e78 subvendor=0x10de subdevice=0x13d8
>>>>> vendor = 'NVIDIA Corporation'
>>>>> device = 'TU102GL [Quadro RTX 6000/8000]'
>>>>> class = display
>>>>> subclass = 3D
>>>>> ppt1@pci0:19:0:0: class=0x030200 rev=0xa1 hdr=0x00
>>>>> vendor=0x10de device=0x1e78 subvendor=0x10de subdevice=0x13d8
>>>>> vendor = 'NVIDIA Corporation'
>>>>> device = 'TU102GL [Quadro RTX 6000/8000]'
>>>>> class = display
>>>>> subclass = 3D
>>>>>
>>>>> as I am using vm-bhyve, I have put the configuration as below
>>>>>
>>>>> # cat /mnt/VMs/jagadish/jagadish.conf
>>>>> loader="uefi"
>>>>> cpu=16
>>>>> memory=128G
>>>>> xhci_mouse="yes"
>>>>> debug="true"
>>>>>
>>>>> graphics="yes"
>>>>> graphics_listen="127.0.0.1"
>>>>> graphics_port="5920"
>>>>> graphics_res="1024x768"
>>>>> graphics_wait="no"
>>>>> #graphics_vga="io"
>>>>>
>>>>> network0_type="virtio-net"
>>>>> network0_switch="swUNI"
>>>>> network0_mac="58:9c:fc:06:3f:af"
>>>>>
>>>>> disk0_type="nvme"
>>>>> disk0_name="jagadish-disk0.img"
>>>>> #disk0_size="128G"
>>>>>
>>>>> uuid="966e909b-1293-11ef-a9a4-e4434bfe34de"
>>>>>
>>>>> passthru0="19/0/0=6:0"
>>>>>
>>>>> bhyve_options="-A -H -P"
>>>>> #END
>>>>>
>>>>> however, when I start the vm (for OS installation to start with), it
>>>>> shows it is running but I am unable to access the VNC for a while, then
>>>>> when I finally manage to connect via VNC, there is nothing on the screen,
>>>>> just blank dark screen
>>>>>
>>>>> the vm-bhyve.log shows
>>>>>
>>>>> Mar 24 13:59:26: initialising
>>>>> Mar 24 13:59:26: [loader: uefi]
>>>>> Mar 24 13:59:26: [cpu: 16]
>>>>> Mar 24 13:59:26: [memory: 128G]
>>>>> Mar 24 13:59:26: [hostbridge: standard]
>>>>> Mar 24 13:59:26: [com ports: com1]
>>>>> Mar 24 13:59:26: [uuid: 966e909b-1293-11ef-a9a4-e4434bfe34de]
>>>>> Mar 24 13:59:26: [debug mode: true]
>>>>> Mar 24 13:59:26: [primary disk: jagadish-disk0.img]
>>>>> Mar 24 13:59:26: [primary disk dev: file]
>>>>> Mar 24 13:59:26: initialising network device tap0
>>>>> Mar 24 13:59:26: adding tap0 -> vm-swUNI (swUNI addm)
>>>>> Mar 24 13:59:26: bring up tap0 -> vm-swUNI (swUNI addm)
>>>>> Mar 24 13:59:26: booting
>>>>> Mar 24 13:59:26: [bhyve options: -c 16 -m 128G -AHPw -l
>>>>> bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -A -H -P -U
>>>>> 966e909b-1293-11ef-a9a4-e4434bfe34de -u -S]
>>>>> Mar 24 13:59:26: [bhyve devices: -s 0,hostbridge -s 31,lpc -s
>>>>> 4:0,nvme,/mnt/VMs/jagadish/jagadish-disk0.img -s
>>>>> 5:0,virtio-net,tap0,mac=58:9c:fc:06:3f:af -s 6:0,passthru,19/0/0 -s
>>>>> 7:0,fbuf,tcp=127.0.0.1:5920,w=1024,h=768 -s 8:0,xhci,tablet]
>>>>> Mar 24 13:59:26: [bhyve console: -l com1,/dev/nmdm-jagadish.1A]
>>>>> Mar 24 13:59:26: [bhyve iso device: -s
>>>>> 3:0,ahci-cd,/mnt/VMs/.iso/lubuntu-24.04.1-desktop-amd64.iso,ro]
>>>>> Mar 24 13:59:26: starting bhyve (run 1)
>>>>>
>>>>> based on what I can see, and a little out of desparation, I decided to
>>>>> run the installer in the foreground, and below is what I got
>>>>>
>>>>> # vm install -f jagadish FreeBSD-14.2-RELEASE-amd64-disc1.iso
>>>>> Starting jagadish
>>>>> * found guest in /mnt/VMs/jagadish
>>>>> * booting...
>>>>> fbuf frame buffer base: 0x112245400000 [sz 16777216]
>>>>>
>>>>> it stays there for as long as I wait until I poweroff the vm.
>>>>>
>>>>> tried with Debian installer,
>>>>> # vm install -f jagadish debian-12.5.0-amd64-netinst.iso
>>>>> Starting jagadish
>>>>> * found guest in /mnt/VMs/jagadish
>>>>> * booting...
>>>>> fbuf frame buffer base: 0x2747e2400000 [sz 16777216]
>>>>>
>>>>>
>>>>> ideally I need to install Debian for my user base on these GPU based
>>>>> systems. But even that is proving to be impossible since the installer does
>>>>> not move any further from where it is.
>>>>>
>>>>> Any thoughts/ideas/suggestions what I else I can try to make this
>>>>> work? Anything that you have tried that worked? have I missed something?
>>>>>
>>>>> Would appreciate any and all thoughts/suggestions
>>>>>
>>>>> best regards
>>>>> SS
>>>>>
>>>>
>>>>
>>>> --
>>>> Mario.
>>>>
>>>
>>>
>>> --
>>> Mario.
>>>
>>
>>
>> --
>> Mario.
>>
>