Re: In response to "GPU Passthrough with FreeBSD 14.2 bhyve"

Good morning Alex, and thank you for the detailed help, it is very much
appreciated.

For some reason your email went into google's spam box, I just noticed it
this morning. I'll try this as well and keep the list posted as to how it
goes.

Best regards
SS

On Tue, 25 Mar 2025 at 13:27, Alex <9vlc@proton.me> wrote:

> This is a hefty help email for Shamim Shahriar in regards to passing
> through Nvidia GPU to a bhyve VM.
>
> Hi.
> I have recently gotten this mailing list message forwarded to me by
> Michael Dexter
> https://lists.freebsd.org/archives/freebsd-virtualization/2025-March/002818.html
> and it alone made me sign up here.
>
> So, here's a big mail containing a bunch of useful GPU passthrough info!
>
> To Shamim, I noticed quite a bit of broken stuff in your vm-bhyve config,
> let me help you out make it work correctly!
>
> Parts and short description:
>
> 1. Pass through the GPU audio and other devices (if exist)
>         Modern GPUs usually create two PCI devices that you can see in
> pciconf - VGA and audio. They come on the same PCI function (i think it is
> called that), for example GPU 19/0/0 will also have 19/0/1 for audio and
> may have 19/0/2 for usb.
>
> 2. Give the GPU a VGA BIOS as PCI Option ROM
>         GPUs need a VGA BIOS to work. In addition, Nvidia GPUs need a
> trimmed VGA BIOS since the PCI option rom that is used by the VM is nested
> inside the ROM file.
>         (from now on i will refer to the vga bios as vbios)
>
> 3. Possibly, re-BAR in BIOS?
>         Some UEFI firmware have a feature called re-BAR and "above 4G
> decoding / mining", it makes some GPUs in some configurations not work in
> passthru.
>
> 5. Possibly, Try newer EDK2
>         The edk2-bhyve package (provides UEFI firmware for bhyve) on the
> repos is still using the 2023 release. You can compile a 2025 version with
> my patch: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=285554
>
> Full parts and HOWTOs:
>
> !!!! MISC GPU DEVICES
>
> As decribed before, pass through all the devices that your GPU creates in
> pciconf right after the GPU device on the same PCI function.
> All audio, USB, etc. devices belonging to the GPU should be passed through
> with it since they are a part of it, else the passthru will NOT work
> correctly.
> If that is not the case and there is only a single GPU VGA device with
> nothing else Nvidia to be found, skip this step.
>
> !!!! VBIOS
>
> You first need to get the pure vBIOS file. To do that you can either:
> 1. Use a Linux system and dump it with
> https://github.com/ilya-zlobintsev/LACT
> 2. Use a Windows system and dump it with https://www.techpowerup.com/gpuz
> 3. Download one that someone has dumped from their GPU using GPU-z from
> https://www.techpowerup.com/vgabios
>
> If you don't want to install a different OS on the PC you're currently
> running, we can just find and download a correct vBIOS from Techpowerup.
> But we stumble across an issue! You specified "NVidia Quadro RTX
> 6000/8000", so which GPU is it really? Those are two different GPUs with
> different specifications!
> Thankfully, we have your pciconf output which says "subdevice=0x13d8" and
> if we refer to Nvidias documentation, on page 7 over here:
>
> https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/quadro-product-literature/NVIDIA-Quadro-RTX-8000-PCIe-Server-Card-PB-FINAL-1219.pdf
> we can see that it is a RTX 8000!
> Here is an unverified but most likely working vBIOS:
> https://www.techpowerup.com/vgabios/256271/256271
>
> Now after we've gotten the vBIOS, we need to trim the start of the file to
> get to the embedded PCI Option ROM.
> For that we will use my tool "trim_nvidia_vbios" from
> https://github.com/9vlc/ptnotes located in progs/trim_nvidia_vbios
> Compile and run the resulting executable with the first argument being the
> downloaded vBIOS and the second argument being an output PCI OpROM. Example:
> $ cc -o trim.elf ptnotes/progs/trim_nvidia_vbios/trim.c
> $ ./trim.elf techpowerup.rom trimmed-vbios.rom
>
> And we come upon a road block: vm-bhyve does not support passing a PCI
> Option ROMs in "passthru=", so we're going to use "bhyve_options=" instead
> to define the GPU VGA device:
> bhyve_options="-s 6:0,passthru,19/0/0,rom=</path/to/trimmed-vbios.rom>"
> Other devices that do not require a PCI Option ROM can be added with
> "passthru=". These devices include the rest of the GPU pci functions like
> audio and usb.
>
> !!!! RE-BAR
>
> If after everything you do get the GPU to work in the VM at least in the
> edk2 firmware,
> Go to your regular UEFI BIOS setup utility and search for options titled
> "BAR", "PCI Re-size BAR", "re-BAR", "Above 4G Encoding" or "Above 4G for
> Mining".
> If you do see re-BAR but cannot disable it, you could try using
> https://github.com/DavidS95/Smokeless_UMAF to modify locked UEFI
> settings. Be warned though, smokeless is a heck of an unstable MESS.
>
> !!!! UPDATED EDK2
>
> As noted previously, EDK2 on FreeBSD ports is pretty outdated so in some
> rare cases, that might cause issues.
> I have submitted a package update bug report for sysutils/edk2 to FreeBSD
> bugzilla over here:
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=285554
> You can download the patch, apply it to your ports tree, then compile and
> install the edk2 package with "FLAVOR=bhyve" to get the fresh UEFI firmware.
>
> ~~~~
>
> Hope I didn't forget anything!
> I wish you success in the GPU passthrough journey!!
> - Alexey
>
>
>