Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro)
- In reply to: Jonathan Vasquez : "Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro)"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Sun, 31 Aug 2025 07:43:32 UTC
On 8/31/25 02:10, Jonathan Vasquez wrote: > Hey all, > > I've updated my blog post with more information and some useful > screenshots: > https://xyinn.org/blog/freebsd/freebsd_bhyve_gpu_passthrough_amd > > The important stuff to mention is that I've also gotten a chance to > play with vm-bhyve, but for the purposes of making sure we get as > clear information as possible without having to unwrap the vm-bhyve > scripts, I'll continue to use bhyve directly. With that said, some > useful info: > > * The vBIOS is not required when running on Windows as mentioned by > Corvin here: > https://lists.freebsd.org/archives/freebsd-virtualization/2022-June/000797.html, > thus I've stopped using it. I was however also able to > successfully use his "echo 1 ..." recommendation to dump the bios > from within Linux (Fedora 42). I actually failed to extract the > bios before (without using amdvbflash) because most likely my > Linux Mint 22.1's Kernel was too old to extract the ROM. When I > tried to do the same exact thing today with Fedora 42, it worked, > and I was able to see the "rom" file from Fedora 42, but not from > Linux Mint 22.1. > * I was able to get the GPU to load without a Code 43 error. But > it's still unusable. I was actually able to get to this state > before but it was inconsistent. Through more research I was able > to get it to be reproducible. There are a few things to mention here: > o The card will always start in a Code 43 failed state > regardless of what I mention below (including if I reboot the > host). > o If you use RDP, the card will always be in a Code 43 state. > o If I use VNC, the card will start in a failed state (as > mentioned above), HOWEVER, you will be able to go into the > Device Manager, Disable the card, and then Enable the card, > and it will be enabled successfully as you can see in my > screenshots. The OS will also give it resources from what we > can see in the Device's Properties' Resources Tab. > > > Given what I mentioned above, I would recommend (as some other people > recommended in other links I've found), to disable the card in the VM > before you shut it down, and then enable it from inside of Windows > when you turn it back on. Although I don't know how necessary this is > given that if I use VNC, even if it starts off in a failed state > (let's say from a fresh host reboot), doing the "device re-nabling > trick" will let it load successfully again. At this point since I've > gotten the card to say it worked, I'm not sure how to tell Windows to > use that as the main GPU when I run my apps. I've looked at some > examples online of using display settings to change the preferred > card, but I don't see it listed there. Am I suppose to see the card in > dxdiag if I'm using VNC? I'm just running the bhyve VM on my server, > and I VNC over from my laptop on the same LAN network. I'm guessing if > things are working correctly I should be able to do that right? I know > there are also some benchmarking apps that show you what GPUs its > detected and maybe allow you to pick one to run a benchmark on. I > tried opening FurMark but it fails saying that it only detects Open GL > 1.1 and all of that... I actually tried installing Mesa for windows to > see what happens and that does help but it's slow and it's clearly not > using the card and most likely doing some sort of software rendering > (maybe like a Windows llvmpipe). > > Jonathan Vasquez > PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 > Sent with ProtonMail Secure Email > > > On Monday, August 25th, 2025 at 14:51, Jonathan Vasquez > <jon@xyinn.org> wrote: >> No go for my testing inside a Linux Mint 22.1 VM. Fully updated, and >> passing in both 3/0/0 and 3/0/1 for my AMD Radeon card. I was able to >> extract some of the "dmesg" and "eglinfo" output. Running "glxinfo" >> also showed that the OpenGL renderer was set to llvmpipe, so software >> rendering :(. "lspci -k" detected it and showed that it was using the >> "amdgpu" kernel driver. Since I'm using vm-bhyve atm I didn't test >> this with the dumped rom, just directly passing them through. I've >> uploaded some of the logs here for now in case anyone wants to take a >> glance at them: >> >> https://xyinn.org/freebsd/files/gpu_pass/2025-08-25/ >> >> Jonathan Vasquez >> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 >> Sent with ProtonMail Secure Email >> >> >> On Monday, August 25th, 2025 at 14:37, Robert Cocker >> <rcocker87@gmail.com> wrote: >>> Hey group has bhvye core dev baked out an update for this >>> >>> Robert >>> ------------------------------------------------------------------------ >>> *From:* owner-freebsd-virtualization@FreeBSD.org >>> <owner-freebsd-virtualization@FreeBSD.org> on behalf of Jonathan >>> Vasquez <jon@xyinn.org> >>> *Sent:* Monday, August 25, 2025 11:35:31 AM >>> *To:* Jonathan Vasquez <jon@xyinn.org> >>> *Cc:* Stephan Althaus <Stephan.Althaus@Duedinghausen.eu>; >>> virtualization@freebsd.org <virtualization@freebsd.org> >>> *Subject:* Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 >>> XT and Windows 10 Pro) >>> I was able to get a fresh Win 10 Pro VM running through vm-bhyve but >>> no progress. Still same Code 43 error as before. I wasn't able to >>> figure out how to pass the |rom=bios.rom| through vm-bhyve, I >>> didn't see anything about it in the config.sample file and I also >>> tried to bypass that by adding it in bhyve_options but no go, the VM >>> wouldn't even start if I did that. I see there is a "bhyve_devices" >>> that contains all of the slot devices but I don't see that as an >>> adjustable variable based on the config.sample info. I've pasted my >>> current vm-bhyve template below. My next test is to try the same >>> thing inside a Linux guest to see what happens. >>> >>> ---------------- >>> >>> loader="uefi" >>> cpu="4" >>> cpu_sockets="1" >>> cpu_cores="4" >>> cpu_threads="1" >>> memory="16G" >>> wired_memory="yes" >>> utctime="no" >>> ahci_device_limit="8" >>> disk0_type="nvme" >>> disk0_name="disk0.img" >>> network0_type="e1000" >>> network0_switch="public" >>> passthru0="3/0/0=2:0" >>> passthru1="3/0/1=2:1" >>> #bhyve_options="-s 2:0,passthru,3/0/0,rom=bios.rom" >>> >>> # Using RDP post-install. >>> graphics="no" >>> graphics_port="5900" >>> graphics_listen="0.0.0.0" >>> graphics_res="1920x1080" >>> graphics_wait="yes" >>> xhci_mouse="yes" >>> uuid="535d54b5-81cb-11f0-93a2-98b78501bda8" >>> network0_mac="58:9c:fc:04:5a:eb" >>> >>> ---------------- >>> >>> Jonathan Vasquez >>> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 >>> Sent with ProtonMail Secure Email >>> >>> >>> On Monday, August 25th, 2025 at 11:05, Jonathan Vasquez >>> <jon@xyinn.org> wrote: >>>> Unfortunately I don't have an intel machine with an available PCIe >>>> slot. However I'm trying another thing which is to passthrough the >>>> card to a Linux guest. I need to run more experiments but a >>>> preliminary boot of a Linux Mint ISO did show the AMD Radeon RX >>>> 6900 XT card in |lspci| and showed that it was using the |amdgpu| >>>> driver. I then did a basic |glxgears| and it opened up in my VNC >>>> window and showed the FPS counter at over 2000 fps. The only issue >>>> I noticed was when shutting down the live CD environment there were >>>> |amdgpu| failure messages where it was failing to write something >>>> to it. I'm currently redoing my "basic bhyve script" and moving it >>>> over to "vm-bhyve" since it seems a lot of people are using that >>>> instead of the more manual approach. This should hopefully help >>>> simplify things for debugging. If the card works on Linux via >>>> passthrough this would also be useful info and can allow me to see >>>> the performance of Cyperpunk 2077, and some other games (even if >>>> I'm not getting 100% of the card's performance). >>>> >>>> Jonathan Vasquez >>>> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 >>>> Sent with ProtonMail Secure Email >>>> >>>> >>>> On Monday, August 25th, 2025 at 11:00, Stephan Althaus >>>> <Stephan.Althaus@Duedinghausen.eu> wrote: >>>>> On 8/25/25 16:56, Jonathan Vasquez wrote: >>>>>> Hey Stephan, >>>>>> >>>>>> Yup, I've tried passing both 3/0/0 and 3/0/1 and no go, and I >>>>>> also make sure to reboot the host in between runs due to the amd >>>>>> hardware issue. I don't believe 3/0/1 is necessary though based >>>>>> on the examples by Corvin in his talk. He only passed in 3/0/0 >>>>>> (I'm guessing his card also has a 3/0/1 function that he isn't >>>>>> using since that's the audio channel). >>>>>> >>>>>> Jonathan Vasquez >>>>>> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 >>>>>> Sent with ProtonMail Secure Email >>>>>> >>>>>> >>>>>> On Monday, August 25th, 2025 at 10:52, Stephan Althaus >>>>>> <Stephan.Althaus@Duedinghausen.eu> >>>>>> <mailto:Stephan.Althaus@Duedinghausen.eu> wrote: >>>>>>> On 8/25/25 16:47, Stephan Althaus wrote: >>>>>>>> On 8/24/25 23:58, Jonathan Vasquez wrote: >>>>>>>>> Hey all, >>>>>>>>> >>>>>>>>> I've been experimenting with trying to get my server's >>>>>>>>> dedicated AMD Radeon RX 6900 XT working on a bhyve Windows 10 >>>>>>>>> Pro client for gaming purposes. I've been reading a bunch of >>>>>>>>> blog posts, sites, files, and videos (thanks Corvin) to try >>>>>>>>> and piece things together. I've been able to get the VM to >>>>>>>>> recognize the card, but still stuck on the Code 43 issue (with >>>>>>>>> or without using the vBIOS I've extracted from my dedicated >>>>>>>>> card and also ensuring any options needed to be on in the UEFI >>>>>>>>> settings are enabled). This machine is running on an AMD Ryzen >>>>>>>>> 9 7950X 16-Core Processor, so it's a full AMD setup, no Intel >>>>>>>>> stuff. It technically should be relatively "easy" to get AMD >>>>>>>>> working from what I've read (excluding the known AMD Hardware >>>>>>>>> reset issue). I've documented everything I've done so far, and >>>>>>>>> will be documenting more and simplifying the instructions as I >>>>>>>>> make progress. Please let me know if you see any thing weird >>>>>>>>> or anything I could try based on what I've rewritten. I'll try >>>>>>>>> to give it a go and provide feedback. Thank you! >>>>>>>>> >>>>>>>>> https://xyinn.org/blog/freebsd/freebsd_bhyve_gpu_passthrough_amd >>>>>>>>> <https://xyinn.org/blog/freebsd/freebsd_bhyve_gpu_passthrough_amd> >>>>>>>>> >>>>>>>>> Jonathan Vasquez >>>>>>>>> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 >>>>>>>>> Sent with ProtonMail Secure Email >>>>>>>>> >>>>>>>> Hi! >>>>>>>> >>>>>>>> i have a AMD RX6400 GPU setup running on 14.2 with win10 guest, >>>>>>>> host is a intel system. >>>>>>>> >>>>>>>> i think you have to passthrough the 3/0/1 device as well! >>>>>>>> >>>>>>>> The bios.rom is extracted from this card in place ? >>>>>>>> >>>>>>>> >>>>>>>> Regards, >>>>>>>> Stephan >>>>>>>> >>>>>>>> >>>>>>> Ah, and be shure to reboot the whole setup, inclusive host, for >>>>>>> every new test-case, >>>>>>> at least as long at is not guaranteed that we can live without.. >>>>>>> >>>>>>> Regars, >>>>>>> Stephan >>>>>>> >>>>>>> >>>>>> >>>>> Hi! >>>>> >>>>> My only next assumption would be that this only does not work on >>>>> AMD host. >>>>> >>>>> You don't have an intel system to test, you would havedone it, if >>>>> so.. (?) >>>>> >>>>> Stephan >>>>> >>>>> >>>> >>> >> > Hello! I have a successful GPU passthrough of an AMD RX6400 to Win10 on an Intel host. Just want to share some Infos + no problem with RDP here, that's my use case + i used GPU-Z within the vm to check if the gpu is actually useable, it shows GPU frequency > 0 , and OpenGL >1.1 when it is. + i have thr ROM in my bhyve start script, the vm has no problem with it. i don't use vm-bhyve so far. - your win 10 vm is EFI install ? + yes, mesa is software rendering, sort of 'software GPU' It seems to me that the only noticeable difference between our systems is that mine is an intel host.. Regards, Stephan