Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro)

I added a few more screenshots from GPU-Z running inside of the VM, and some other details. Interestingly enough, I was able to dump the VBIOS from inside the VM and the hashes of the dumped VBIOS matches the hash and the file size of the amdvbflash version that I originally dumped from Linux running on this machine in bare metal.. so that means it is possible to dump the VBIOS from within the bhyve VM after you pass the device through. I haven't tried this but it seems that you should also be able to dump the VBIOS from GPU-Z even without the AMD Adrenaline drivers being installed since GPU-Z still detected the card and still showed the option to dump the VBIOS. I only dumped it with the AMD Adrenaline drivers installed.

SHA1: amdvbflash Extracted ROM: 517e09d7ac6be5da0b0fac7d09f782a3c9494343
File Size: 1.0M

SHA1: GPU-Z Extracted ROM: 517e09d7ac6be5da0b0fac7d09f782a3c9494343
File Size: 1.0M

SHA1: Direct Linux Extracted ROM: abbdfd11c246f570839fa7ad78186dcb12408595File Size: 119K

Jonathan Vasquez
PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279
Sent with ProtonMail Secure Email

On Sunday, August 31st, 2025 at 08:00, Jonathan Vasquez <jon@xyinn.org> wrote:

> Thanks for that info Stephan. Yup it's an UEFI install since I'm using the following line in my bhyve script:
>
> -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,fwcfg=qemu
>
> And yea it may be possible something is going on with AMD CPU hosts...
>
> Jonathan Vasquez
> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279
> Sent with ProtonMail Secure Email
>
> Sent with [Proton Mail](https://proton.me/mail/home) secure email.
>
> -------- Original Message --------
> On Saturday, 08/30/25 at 20:11 Jonathan Vasquez <jon@xyinn.org> wrote:
>
>> Hey all,
>>
>> I've updated my blog post with more information and some useful screenshots: https://xyinn.org/blog/freebsd/freebsd_bhyve_gpu_passthrough_amd
>>
>> The important stuff to mention is that I've also gotten a chance to play with vm-bhyve, but for the purposes of making sure we get as clear information as possible without having to unwrap the vm-bhyve scripts, I'll continue to use bhyve directly. With that said, some useful info:
>>
>> - The vBIOS is not required when running on Windows as mentioned by Corvin here: https://lists.freebsd.org/archives/freebsd-virtualization/2022-June/000797.html, thus I've stopped using it. I was however also able to successfully use his "echo 1 ..." recommendation to dump the bios from within Linux (Fedora 42). I actually failed to extract the bios before (without using amdvbflash) because most likely my Linux Mint 22.1's Kernel was too old to extract the ROM. When I tried to do the same exact thing today with Fedora 42, it worked, and I was able to see the "rom" file from Fedora 42, but not from Linux Mint 22.1.
>> - I was able to get the GPU to load without a Code 43 error. But it's still unusable. I was actually able to get to this state before but it was inconsistent. Through more research I was able to get it to be reproducible. There are a few things to mention here:
>>
>> - The card will always start in a Code 43 failed state regardless of what I mention below (including if I reboot the host).
>> - If you use RDP, the card will always be in a Code 43 state.
>> - If I use VNC, the card will start in a failed state (as mentioned above), HOWEVER, you will be able to go into the Device Manager, Disable the card, and then Enable the card, and it will be enabled successfully as you can see in my screenshots. The OS will also give it resources from what we can see in the Device's Properties' Resources Tab.
>>
>> Given what I mentioned above, I would recommend (as some other people recommended in other links I've found), to disable the card in the VM before you shut it down, and then enable it from inside of Windows when you turn it back on. Although I don't know how necessary this is given that if I use VNC, even if it starts off in a failed state (let's say from a fresh host reboot), doing the "device re-nabling trick" will let it load successfully again. At this point since I've gotten the card to say it worked, I'm not sure how to tell Windows to use that as the main GPU when I run my apps. I've looked at some examples online of using display settings to change the preferred card, but I don't see it listed there. Am I suppose to see the card in dxdiag if I'm using VNC? I'm just running the bhyve VM on my server, and I VNC over from my laptop on the same LAN network. I'm guessing if things are working correctly I should be able to do that right? I know there are also some benchmarking apps that show you what GPUs its detected and maybe allow you to pick one to run a benchmark on. I tried opening FurMark but it fails saying that it only detects Open GL 1.1 and all of that... I actually tried installing Mesa for windows to see what happens and that does help but it's slow and it's clearly not using the card and most likely doing some sort of software rendering (maybe like a Windows llvmpipe).
>>
>> Jonathan Vasquez
>> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279
>> Sent with ProtonMail Secure Email
>>
>> On Monday, August 25th, 2025 at 14:51, Jonathan Vasquez <jon@xyinn.org> wrote:
>>
>>> No go for my testing inside a Linux Mint 22.1 VM. Fully updated, and passing in both 3/0/0 and 3/0/1 for my AMD Radeon card. I was able to extract some of the "dmesg" and "eglinfo" output. Running "glxinfo" also showed that the OpenGL renderer was set to llvmpipe, so software rendering :(. "lspci -k" detected it and showed that it was using the "amdgpu" kernel driver. Since I'm using vm-bhyve atm I didn't test this with the dumped rom, just directly passing them through. I've uploaded some of the logs here for now in case anyone wants to take a glance at them:
>>>
>>> https://xyinn.org/freebsd/files/gpu_pass/2025-08-25/
>>>
>>> Jonathan Vasquez
>>> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279
>>> Sent with ProtonMail Secure Email
>>>
>>> On Monday, August 25th, 2025 at 14:37, Robert Cocker <rcocker87@gmail.com> wrote:
>>>
>>>> Hey group has bhvye core dev baked out an update for this
>>>>
>>>> Robert
>>>> ---------------------------------------------------------------
>>>>
>>>> From: owner-freebsd-virtualization@FreeBSD.org <owner-freebsd-virtualization@FreeBSD.org> on behalf of Jonathan Vasquez <jon@xyinn.org>
>>>> Sent: Monday, August 25, 2025 11:35:31 AM
>>>> To: Jonathan Vasquez <jon@xyinn.org>
>>>> Cc: Stephan Althaus <Stephan.Althaus@Duedinghausen.eu>; virtualization@freebsd.org <virtualization@freebsd.org>
>>>> Subject: Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro)
>>>>
>>>> I was able to get a fresh Win 10 Pro VM running through vm-bhyve but no progress. Still same Code 43 error as before. I wasn't able to figure out how to pass the rom=bios.rom​ through vm-bhyve, I didn't see anything about it in the config.sample file and I also tried to bypass that by adding it in bhyve_options but no go, the VM wouldn't even start if I did that. I see there is a "bhyve_devices" that contains all of the slot devices but I don't see that as an adjustable variable based on the config.sample info. I've pasted my current vm-bhyve template below. My next test is to try the same thing inside a Linux guest to see what happens.
>>>>
>>>> ----------------
>>>>
>>>> loader="uefi"
>>>> cpu="4"
>>>> cpu_sockets="1"
>>>> cpu_cores="4"
>>>> cpu_threads="1"
>>>> memory="16G"
>>>> wired_memory="yes"
>>>>
>>>> utctime="no"
>>>> ahci_device_limit="8"
>>>>
>>>> disk0_type="nvme"
>>>> disk0_name="disk0.img"
>>>>
>>>> network0_type="e1000"
>>>> network0_switch="public"
>>>>
>>>> passthru0="3/0/0=2:0"
>>>> passthru1="3/0/1=2:1"
>>>>
>>>> #bhyve_options="-s 2:0,passthru,3/0/0,rom=bios.rom"
>>>>
>>>> # Using RDP post-install.
>>>> graphics="no"
>>>> graphics_port="5900"
>>>> graphics_listen="0.0.0.0"
>>>> graphics_res="1920x1080"
>>>> graphics_wait="yes"
>>>>
>>>> xhci_mouse="yes"
>>>> uuid="535d54b5-81cb-11f0-93a2-98b78501bda8"  network0_mac="58:9c:fc:04:5a:eb"
>>>>
>>>> ----------------
>>>>
>>>> Jonathan Vasquez
>>>> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279
>>>> Sent with ProtonMail Secure Email
>>>>
>>>> On Monday, August 25th, 2025 at 11:05, Jonathan Vasquez <jon@xyinn.org> wrote:
>>>>
>>>>> Unfortunately I don't have an intel machine with an available PCIe slot. However I'm trying another thing which is to passthrough the card to a Linux guest. I need to run more experiments but a preliminary boot of a Linux Mint ISO did show the AMD Radeon RX 6900 XT card in lspci​ and showed that it was using the amdgpu​ driver. I then did a basic glxgears​ and it opened up in my VNC window and showed the FPS counter at over 2000 fps. The only issue I noticed was when shutting down the live CD environment there were amdgpu​ failure messages where it was failing to write something to it. I'm currently redoing my "basic bhyve script" and moving it over to "vm-bhyve" since it seems a lot of people are using that instead of the more manual approach. This should hopefully help simplify things for debugging. If the card works on Linux via passthrough this would also be useful info and can allow me to see the performance of Cyperpunk 2077, and some other games (even if I'm not getting 100% of the card's performance).
>>>>>
>>>>> Jonathan Vasquez
>>>>> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279
>>>>> Sent with ProtonMail Secure Email
>>>>>
>>>>> On Monday, August 25th, 2025 at 11:00, Stephan Althaus <Stephan.Althaus@Duedinghausen.eu> wrote:
>>>>>
>>>>>> On 8/25/25 16:56, Jonathan Vasquez wrote:
>>>>>>
>>>>>>> Hey Stephan,
>>>>>>>
>>>>>>> Yup, I've tried passing both 3/0/0 and 3/0/1 and no go, and I also make sure to reboot the host in between runs due to the amd hardware issue. I don't believe 3/0/1 is necessary though based on the examples by Corvin in his talk. He only passed in 3/0/0 (I'm guessing his card also has a 3/0/1 function that he isn't using since that's the audio channel).
>>>>>>>
>>>>>>> Jonathan Vasquez
>>>>>>> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279
>>>>>>> Sent with ProtonMail Secure Email
>>>>>>>
>>>>>>> On Monday, August 25th, 2025 at 10:52, Stephan Althaus <Stephan.Althaus@Duedinghausen.eu> wrote:
>>>>>>>
>>>>>>>> On 8/25/25 16:47, Stephan Althaus wrote:
>>>>>>>>
>>>>>>>>> On 8/24/25 23:58, Jonathan Vasquez wrote:
>>>>>>>>>
>>>>>>>>>> Hey all,
>>>>>>>>>>
>>>>>>>>>> I've been experimenting with trying to get my server's dedicated AMD Radeon RX 6900 XT working on a bhyve Windows 10 Pro client for gaming purposes. I've been reading a bunch of blog posts, sites, files, and videos (thanks Corvin) to try and piece things together. I've been able to get the VM to recognize the card, but still stuck on the Code 43 issue (with or without using the vBIOS I've extracted from my dedicated card and also ensuring any options needed to be on in the UEFI settings are enabled). This machine is running on an AMD Ryzen 9 7950X 16-Core Processor, so it's a full AMD setup, no Intel stuff. It technically should be relatively "easy" to get AMD working from what I've read (excluding the known AMD Hardware reset issue). I've documented everything I've done so far, and will be documenting more and simplifying the instructions as I make progress. Please let me know if you see any thing weird or anything I could try based on what I've rewritten. I'll try to give it a go and provide feedback. Thank you!
>>>>>>>>>>
>>>>>>>>>> https://xyinn.org/blog/freebsd/freebsd_bhyve_gpu_passthrough_amd
>>>>>>>>>>
>>>>>>>>>> Jonathan Vasquez
>>>>>>>>>> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279
>>>>>>>>>> Sent with ProtonMail Secure Email
>>>>>>>>>
>>>>>>>>> Hi!
>>>>>>>>>
>>>>>>>>> i have a AMD RX6400 GPU setup running on 14.2 with win10 guest, host is a intel system.
>>>>>>>>>
>>>>>>>>> i think you have to passthrough the 3/0/1 device as well!
>>>>>>>>>
>>>>>>>>> The bios.rom is extracted from this card in place ?
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Stephan
>>>>>>>>
>>>>>>>> Ah, and be shure to reboot the whole setup, inclusive host, for every new test-case,
>>>>>>>> at least as long at is not guaranteed that we can live without..
>>>>>>>>
>>>>>>>> Regars,
>>>>>>>> Stephan
>>>>>>
>>>>>> Hi!
>>>>>>
>>>>>> My only next assumption would be that this only does not work on AMD host.
>>>>>>
>>>>>> You don't have an intel system to test, you would havedone it, if so.. (?)
>>>>>>
>>>>>> Stephan