Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro)

Reply: Tomek CEDRO : "Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro)"
Reply: Corvin K�hne : "Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro)"
Go to: [ bottom of page ] [ top of archives ] [ this month ]
From: Jonathan Vasquez <jon_at_xyinn.org>
Date: Tue, 16 Sep 2025 20:29:19 UTC
Hello everyone,

I wanted to update you on my progress on this. After some discussion with Corvin, I was able to use some of his insights to finally get it working :). There was only one thing that was off with my configuration that was causing it not to work at all, and that's that the "Resize BAR Support" option was enabled in my UEFI. Apparently FreeBSD doesn't support Resizable BAR and if this is enabled, then we won't know what will happen exactly if the host or the bhyve VM decides to resize the bar. After I turned that off, everything worked! I've done a full update of my instructions on how to get this working and I've also generated a new set of screenshots of multiple different games and benchmarks. Let me know your thoughts. I'm not familiar with Resize BAR, but my BIOS mentioned that it's to allow you "take full advantage of the GPUs memory", I'm guessing that there may be a significant performance drop due to this being disabled, but the performance is already pretty good. I was getting 40 fps pretty much consistently in Cyberpunk 2077 on Ray Tracing Low on 2560x1440p. The Heaven Benchmarks were getting 100-300 fps depending on the scene on DX11, Ultra Quality, Extreme Tessellation. Furmark also yielded 194-208 (avg: 201) fps for its 1440p benchmark test. This was all through RDP.

Thanks for answering my questions Corvin, much appreciated :).

Blog post can be found here: https://xyinn.org/blog/freebsd/freebsd_bhyve_gpu_passthrough_amd

I also wanted to point out the "size" of Bar 10 and 18 before I disabled Resize BAR support in the BIOS and afterwards:

Resize BAR Support - Enabled

# pciconf -lbevV pci0:3:0:0
ppt0@pci0:3:0:0:        class=0x030000 rev=0xc0 hdr=0x00 vendor=0x1002 device=0x73af subvendor=0x1043 subdevice=0x04fe
    vendor     = 'Advanced Micro Devices, Inc. [AMD/ATI]'
    device     = 'Navi 21 [Radeon RX 6900 XT]'
    class      = display
    subclass   = VGA
    bar   [10] = type Prefetchable Memory, range 64, base 0xf800000000, size 17179869184, disabled
    bar   [18] = type Prefetchable Memory, range 64, base 0xfc00000000, size 268435456, disabled
    bar   [20] = type I/O Port, range 32, base 0xf000, size 256, disabled
    bar   [24] = type Memory, range 32, base 0xf6a00000, size 1048576, disabled
  PCI-e errors = Correctable Error Detected
                 Unsupported Request Detected
     Corrected = Advisory Non-Fatal Error

Resize BAR Support - Disabled

# pciconf -lbevV pci0:3:0:0
ppt0@pci0:3:0:0:        class=0x030000 rev=0xc0 hdr=0x00 vendor=0x1002 device=0x73af subvendor=0x1043 subdevice=0x04fe
    vendor     = 'Advanced Micro Devices, Inc. [AMD/ATI]'
    device     = 'Navi 21 [Radeon RX 6900 XT]'
    class      = display
    subclass   = VGA
    bar   [10] = type Prefetchable Memory, range 64, base 0xfce0000000, size 268435456, disabled
    bar   [18] = type Prefetchable Memory, range 64, base 0xfcf0000000, size 2097152, disabled
    bar   [20] = type I/O Port, range 32, base 0xf000, size 256, disabled
    bar   [24] = type Memory, range 32, base 0xf6a00000, size 1048576, disabled
  PCI-e errors = Correctable Error Detected
                 Unsupported Request Detected
     Corrected = Advisory Non-Fatal Error

On Sunday, August 31st, 2025 at 10:18, Jonathan Vasquez <jon@xyinn.org> wrote:

> I added a few more screenshots from GPU-Z running inside of the VM, and some other details. Interestingly enough, I was able to dump the VBIOS from inside the VM and the hashes of the dumped VBIOS matches the hash and the file size of the amdvbflash version that I originally dumped from Linux running on this machine in bare metal.. so that means it is possible to dump the VBIOS from within the bhyve VM after you pass the device through. I haven't tried this but it seems that you should also be able to dump the VBIOS from GPU-Z even without the AMD Adrenaline drivers being installed since GPU-Z still detected the card and still showed the option to dump the VBIOS. I only dumped it with the AMD Adrenaline drivers installed.
>
> SHA1: amdvbflash Extracted ROM: 517e09d7ac6be5da0b0fac7d09f782a3c9494343
> File Size: 1.0M
>
> SHA1: GPU-Z Extracted ROM: 517e09d7ac6be5da0b0fac7d09f782a3c9494343
> File Size: 1.0M
>
> SHA1: Direct Linux Extracted ROM: abbdfd11c246f570839fa7ad78186dcb12408595File Size: 119K
>
> Jonathan Vasquez
> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279
> Sent with ProtonMail Secure Email
>
> On Sunday, August 31st, 2025 at 08:00, Jonathan Vasquez <jon@xyinn.org> wrote:
>
>> Thanks for that info Stephan. Yup it's an UEFI install since I'm using the following line in my bhyve script:
>>
>> -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,fwcfg=qemu
>>
>> And yea it may be possible something is going on with AMD CPU hosts...
>>
>> Jonathan Vasquez
>> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279
>> Sent with ProtonMail Secure Email
>>
>> Sent with [Proton Mail](https://proton.me/mail/home) secure email.
>>
>> -------- Original Message --------
>> On Saturday, 08/30/25 at 20:11 Jonathan Vasquez <jon@xyinn.org> wrote:
>>
>>> Hey all,
>>>
>>> I've updated my blog post with more information and some useful screenshots: https://xyinn.org/blog/freebsd/freebsd_bhyve_gpu_passthrough_amd
>>>
>>> The important stuff to mention is that I've also gotten a chance to play with vm-bhyve, but for the purposes of making sure we get as clear information as possible without having to unwrap the vm-bhyve scripts, I'll continue to use bhyve directly. With that said, some useful info:
>>>
>>> - The vBIOS is not required when running on Windows as mentioned by Corvin here: https://lists.freebsd.org/archives/freebsd-virtualization/2022-June/000797.html, thus I've stopped using it. I was however also able to successfully use his "echo 1 ..." recommendation to dump the bios from within Linux (Fedora 42). I actually failed to extract the bios before (without using amdvbflash) because most likely my Linux Mint 22.1's Kernel was too old to extract the ROM. When I tried to do the same exact thing today with Fedora 42, it worked, and I was able to see the "rom" file from Fedora 42, but not from Linux Mint 22.1.
>>> - I was able to get the GPU to load without a Code 43 error. But it's still unusable. I was actually able to get to this state before but it was inconsistent. Through more research I was able to get it to be reproducible. There are a few things to mention here:
>>>
>>> - The card will always start in a Code 43 failed state regardless of what I mention below (including if I reboot the host).
>>> - If you use RDP, the card will always be in a Code 43 state.
>>> - If I use VNC, the card will start in a failed state (as mentioned above), HOWEVER, you will be able to go into the Device Manager, Disable the card, and then Enable the card, and it will be enabled successfully as you can see in my screenshots. The OS will also give it resources from what we can see in the Device's Properties' Resources Tab.
>>>
>>> Given what I mentioned above, I would recommend (as some other people recommended in other links I've found), to disable the card in the VM before you shut it down, and then enable it from inside of Windows when you turn it back on. Although I don't know how necessary this is given that if I use VNC, even if it starts off in a failed state (let's say from a fresh host reboot), doing the "device re-nabling trick" will let it load successfully again. At this point since I've gotten the card to say it worked, I'm not sure how to tell Windows to use that as the main GPU when I run my apps. I've looked at some examples online of using display settings to change the preferred card, but I don't see it listed there. Am I suppose to see the card in dxdiag if I'm using VNC? I'm just running the bhyve VM on my server, and I VNC over from my laptop on the same LAN network. I'm guessing if things are working correctly I should be able to do that right? I know there are also some benchmarking apps that show you what GPUs its detected and maybe allow you to pick one to run a benchmark on. I tried opening FurMark but it fails saying that it only detects Open GL 1.1 and all of that... I actually tried installing Mesa for windows to see what happens and that does help but it's slow and it's clearly not using the card and most likely doing some sort of software rendering (maybe like a Windows llvmpipe).
>>>
>>> Jonathan Vasquez
>>> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279
>>> Sent with ProtonMail Secure Email
>>>
>>> On Monday, August 25th, 2025 at 14:51, Jonathan Vasquez <jon@xyinn.org> wrote:
>>>
>>>> No go for my testing inside a Linux Mint 22.1 VM. Fully updated, and passing in both 3/0/0 and 3/0/1 for my AMD Radeon card. I was able to extract some of the "dmesg" and "eglinfo" output. Running "glxinfo" also showed that the OpenGL renderer was set to llvmpipe, so software rendering :(. "lspci -k" detected it and showed that it was using the "amdgpu" kernel driver. Since I'm using vm-bhyve atm I didn't test this with the dumped rom, just directly passing them through. I've uploaded some of the logs here for now in case anyone wants to take a glance at them:
>>>>
>>>> https://xyinn.org/freebsd/files/gpu_pass/2025-08-25/
>>>>
>>>> Jonathan Vasquez
>>>> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279
>>>> Sent with ProtonMail Secure Email
>>>>
>>>> On Monday, August 25th, 2025 at 14:37, Robert Cocker <rcocker87@gmail.com> wrote:
>>>>
>>>>> Hey group has bhvye core dev baked out an update for this
>>>>>
>>>>> Robert
>>>>> ---------------------------------------------------------------
>>>>>
>>>>> From: owner-freebsd-virtualization@FreeBSD.org <owner-freebsd-virtualization@FreeBSD.org> on behalf of Jonathan Vasquez <jon@xyinn.org>
>>>>> Sent: Monday, August 25, 2025 11:35:31 AM
>>>>> To: Jonathan Vasquez <jon@xyinn.org>
>>>>> Cc: Stephan Althaus <Stephan.Althaus@Duedinghausen.eu>; virtualization@freebsd.org <virtualization@freebsd.org>
>>>>> Subject: Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro)
>>>>>
>>>>> I was able to get a fresh Win 10 Pro VM running through vm-bhyve but no progress. Still same Code 43 error as before. I wasn't able to figure out how to pass the rom=bios.rom through vm-bhyve, I didn't see anything about it in the config.sample file and I also tried to bypass that by adding it in bhyve_options but no go, the VM wouldn't even start if I did that. I see there is a "bhyve_devices" that contains all of the slot devices but I don't see that as an adjustable variable based on the config.sample info. I've pasted my current vm-bhyve template below. My next test is to try the same thing inside a Linux guest to see what happens.
>>>>>
>>>>> ----------------
>>>>>
>>>>> loader="uefi"
>>>>> cpu="4"
>>>>> cpu_sockets="1"
>>>>> cpu_cores="4"
>>>>> cpu_threads="1"
>>>>> memory="16G"
>>>>> wired_memory="yes"
>>>>>
>>>>> utctime="no"
>>>>> ahci_device_limit="8"
>>>>>
>>>>> disk0_type="nvme"
>>>>> disk0_name="disk0.img"
>>>>>
>>>>> network0_type="e1000"
>>>>> network0_switch="public"
>>>>>
>>>>> passthru0="3/0/0=2:0"
>>>>> passthru1="3/0/1=2:1"
>>>>>
>>>>> #bhyve_options="-s 2:0,passthru,3/0/0,rom=bios.rom"
>>>>>
>>>>> # Using RDP post-install.
>>>>> graphics="no"
>>>>> graphics_port="5900"
>>>>> graphics_listen="0.0.0.0"
>>>>> graphics_res="1920x1080"
>>>>> graphics_wait="yes"
>>>>>
>>>>> xhci_mouse="yes"
>>>>> uuid="535d54b5-81cb-11f0-93a2-98b78501bda8"  network0_mac="58:9c:fc:04:5a:eb"
>>>>>
>>>>> ----------------
>>>>>
>>>>> Jonathan Vasquez
>>>>> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279
>>>>> Sent with ProtonMail Secure Email
>>>>>
>>>>> On Monday, August 25th, 2025 at 11:05, Jonathan Vasquez <jon@xyinn.org> wrote:
>>>>>
>>>>>> Unfortunately I don't have an intel machine with an available PCIe slot. However I'm trying another thing which is to passthrough the card to a Linux guest. I need to run more experiments but a preliminary boot of a Linux Mint ISO did show the AMD Radeon RX 6900 XT card in lspci and showed that it was using the amdgpu driver. I then did a basic glxgears and it opened up in my VNC window and showed the FPS counter at over 2000 fps. The only issue I noticed was when shutting down the live CD environment there were amdgpu failure messages where it was failing to write something to it. I'm currently redoing my "basic bhyve script" and moving it over to "vm-bhyve" since it seems a lot of people are using that instead of the more manual approach. This should hopefully help simplify things for debugging. If the card works on Linux via passthrough this would also be useful info and can allow me to see the performance of Cyperpunk 2077, and some other games (even if I'm not getting 100% of the card's performance).
>>>>>>
>>>>>> Jonathan Vasquez
>>>>>> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279
>>>>>> Sent with ProtonMail Secure Email
>>>>>>
>>>>>> On Monday, August 25th, 2025 at 11:00, Stephan Althaus <Stephan.Althaus@Duedinghausen.eu> wrote:
>>>>>>
>>>>>>> On 8/25/25 16:56, Jonathan Vasquez wrote:
>>>>>>>
>>>>>>>> Hey Stephan,
>>>>>>>>
>>>>>>>> Yup, I've tried passing both 3/0/0 and 3/0/1 and no go, and I also make sure to reboot the host in between runs due to the amd hardware issue. I don't believe 3/0/1 is necessary though based on the examples by Corvin in his talk. He only passed in 3/0/0 (I'm guessing his card also has a 3/0/1 function that he isn't using since that's the audio channel).
>>>>>>>>
>>>>>>>> Jonathan Vasquez
>>>>>>>> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279
>>>>>>>> Sent with ProtonMail Secure Email
>>>>>>>>
>>>>>>>> On Monday, August 25th, 2025 at 10:52, Stephan Althaus <Stephan.Althaus@Duedinghausen.eu> wrote:
>>>>>>>>
>>>>>>>>> On 8/25/25 16:47, Stephan Althaus wrote:
>>>>>>>>>
>>>>>>>>>> On 8/24/25 23:58, Jonathan Vasquez wrote:
>>>>>>>>>>
>>>>>>>>>>> Hey all,
>>>>>>>>>>>
>>>>>>>>>>> I've been experimenting with trying to get my server's dedicated AMD Radeon RX 6900 XT working on a bhyve Windows 10 Pro client for gaming purposes. I've been reading a bunch of blog posts, sites, files, and videos (thanks Corvin) to try and piece things together. I've been able to get the VM to recognize the card, but still stuck on the Code 43 issue (with or without using the vBIOS I've extracted from my dedicated card and also ensuring any options needed to be on in the UEFI settings are enabled). This machine is running on an AMD Ryzen 9 7950X 16-Core Processor, so it's a full AMD setup, no Intel stuff. It technically should be relatively "easy" to get AMD working from what I've read (excluding the known AMD Hardware reset issue). I've documented everything I've done so far, and will be documenting more and simplifying the instructions as I make progress. Please let me know if you see any thing weird or anything I could try based on what I've rewritten. I'll try to give it a go and provide feedback. Thank you!
>>>>>>>>>>>
>>>>>>>>>>> https://xyinn.org/blog/freebsd/freebsd_bhyve_gpu_passthrough_amd
>>>>>>>>>>>
>>>>>>>>>>> Jonathan Vasquez
>>>>>>>>>>> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279
>>>>>>>>>>> Sent with ProtonMail Secure Email
>>>>>>>>>>
>>>>>>>>>> Hi!
>>>>>>>>>>
>>>>>>>>>> i have a AMD RX6400 GPU setup running on 14.2 with win10 guest, host is a intel system.
>>>>>>>>>>
>>>>>>>>>> i think you have to passthrough the 3/0/1 device as well!
>>>>>>>>>>
>>>>>>>>>> The bios.rom is extracted from this card in place ?
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Stephan
>>>>>>>>>
>>>>>>>>> Ah, and be shure to reboot the whole setup, inclusive host, for every new test-case,
>>>>>>>>> at least as long at is not guaranteed that we can live without..
>>>>>>>>>
>>>>>>>>> Regars,
>>>>>>>>> Stephan
>>>>>>>
>>>>>>> Hi!
>>>>>>>
>>>>>>> My only next assumption would be that this only does not work on AMD host.
>>>>>>>
>>>>>>> You don't have an intel system to test, you would havedone it, if so.. (?)
>>>>>>>
>>>>>>> Stephan