Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro)
- In reply to: Jonathan Vasquez : "Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro)"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Mon, 22 Sep 2025 06:48:17 UTC
what i noticed that the new pc have very complex bios setup options that may impact stability of freebsd. i encountered this recently after hw upgrade. pci-e gen 4 or 5, x16 or x8 mode, and split controller between single x16 for gpu or x8 for gpu and x8/x4 between other peripherals such as nvme controller usb additional card slots etc. in addition to that there are power modes and runtime frequency changes for cpu and all peripherals. not all new features are supported by freebsd. it took me some while to adjust optimal settings to get machine stable (at the cost of power savings). also there were irritating glitches in audio that are now considerably less noticable. long story short it is cpu, ram, and peripherals settings that may impact overall stability of the freebsd box. this work seems to be not only important for virtualization but also FreeBSD's LDWG project (Laptop Desktop Work Group) i am attaching freebsd-desktop@ mailing list :-) -- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info On Mon, Sep 22, 2025, 02:20 Jonathan Vasquez <jon@xyinn.org> wrote: > Hey all, > > Just wanted to report that I still haven't received one crash after my > adjustments yesterday. So far over 20+ hours of uptime~. I've also rebooted > the VM/host a few times just to ensure that it is deterministically stable > and so far zero issues. So this is pointing in a positive direction. I'm > tempted to mark this as a stable set up but I'll need to play for longer to > make that call. > > Jonathan Vasquez > PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 > Sent with ProtonMail Secure Email > > > On Sunday, September 21st, 2025 at 08:23, Jonathan Vasquez <jon@xyinn.org> > wrote: > > System has been up for about 9h40m. Most of it idle and no crashes yet. > I'm gonna reboot the host and see if the stability still continues. > > Jonathan Vasquez > PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 > Sent with ProtonMail Secure Email > > > > Sent from Proton Mail <https://proton.me/mail/home> for Android. > > > -------- Original Message -------- > On Saturday, 09/20/25 at 23:45 Jonathan Vasquez <jon@xyinn.org> wrote: > > Just finished beating the main story line in Cyberpunk 2077 in the VM. So > far no crashes and it's been running for an hour. So this is a positive > signal... we'll see what happens. I'll leave the VM running idle over night > and see if it's still running in the morning. Stay safe all! > > Jonathan Vasquez > PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 > Sent with ProtonMail Secure Email > > > On Saturday, September 20th, 2025 at 22:28, Jonathan Vasquez < > jon@xyinn.org> wrote: > > I did a bunch of more experiments today and I was able to switch from > using the USB controller connected to the multi-functional bus that has the > integrated card, and switched to a separate USB controller that only has > USB ports. I'm hoping that this increases VM stability and that there may > have been some wires being crossed. This was the previous bus I was using > (I couldn't isolate just the USB functions, it had to be all or nothing for > it to function): > > vgapci0@pci0:18:0:0: class=0x030000 rev=0xc1 hdr=0x00 vendor=0x1002 > device=0x164e subvendor=0x1043 subdevice=0x8877 > vendor = 'Advanced Micro Devices, Inc. [AMD/ATI]' > device = 'Raphael' > class = display > subclass = VGA > hdac0@pci0:18:0:1: class=0x040300 rev=0x00 hdr=0x00 vendor=0x1002 > device=0x1640 subvendor=0x1043 subdevice=0x8877 > vendor = 'Advanced Micro Devices, Inc. [AMD/ATI]' > device = 'Rembrandt Radeon High Definition Audio Controller' > class = multimedia > subclass = HDA > none1@pci0:18:0:2: class=0x108000 rev=0x00 hdr=0x00 vendor=0x1022 > device=0x1649 subvendor=0x1043 subdevice=0x8877 > vendor = 'Advanced Micro Devices, Inc. [AMD]' > device = 'Family 19h PSP/CCP' > class = encrypt/decrypt > xhci1@pci0:18:0:3: class=0x0c0330 rev=0x00 hdr=0x00 vendor=0x1022 > device=0x15b6 subvendor=0x1043 subdevice=0x8877 > vendor = 'Advanced Micro Devices, Inc. [AMD]' > device = 'Raphael/Granite Ridge USB 3.1 xHCI' > class = serial bus > subclass = USB > xhci2@pci0:18:0:4: class=0x0c0330 rev=0x00 hdr=0x00 vendor=0x1022 > device=0x15b7 subvendor=0x1043 subdevice=0x8877 > vendor = 'Advanced Micro Devices, Inc. [AMD]' > device = 'Raphael/Granite Ridge USB 3.1 xHCI' > class = serial bus > subclass = USB > > Although now I'm longer using this at all for USB functionality in the VM. > I've switched to: > > ppt2@pci0:13:0:0: class=0x0c0330 rev=0x01 hdr=0x00 vendor=0x1022 > device=0x43f7 subvendor=0x1b21 subdevice=0x1142 > vendor = 'Advanced Micro Devices, Inc. [AMD]' > device = '600 Series Chipset USB 3.2 Controller' > class = serial bus > subclass = USB > > Much cleaner and still contains a variety of high speed ports. Although > ultimately I'm just using one of them because I have a KVM connected to it > that allows me to plug in keyboard/mouse/3.5mm jack (audio)/ethernet all > through one USB wire. Although to remind everyone, I'm gaming in an offline > capacity to I intentionally mess up the IPv4 gateway so my VM can only > communicate internally (and I can sideload apps/games) but prevent > Microsoft from reaching my machine and collecting telemetry info, and doing > forced updates. Plus, I have no intention of moving to Windows 11 with all > of their new requirements and forced online activation (I know of the > workarounds, but I'm against on mandating a MSA on principle). People > should be able to fully use their machines even during installation/OOBE > without having to authenticate to a central server. Anyways, small rant > over lol. > > I'll update everyone on my progress with this re-configuration. > > Jonathan Vasquez > PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 > Sent with ProtonMail Secure Email > > > On Saturday, September 20th, 2025 at 20:02, Jonathan Vasquez < > jon@xyinn.org> wrote: > > Hey Nuno, > > > 1. I'm not. I plugged in my monitor's HDMI cable directly into the > GPU's HDMI port. Once bhyve starts up (if your passthrough is working > correctly and your GPU's drivers are installed), it will output to your > monitor. You would only need VNC for initial installation of Windows. > 2. I'm not sure of the NVIDIA case. I do know there is someone who > blogged about how they got theirs working: > https://dflund.se/~getz/Notes/2024/freebsd-gpu/ > > > Atm I'm currently trying to collect more info on why my bhyve VM randomly > crashes sometimes. So far I've collected that the exit reason bhyve is > showing is 0x60, which points to the following entries in the amd freebsd > kernel sources: > > 0x60 > > https://github.com/freebsd/freebsd-src/blob/4c3a868d13c053ef173268cdfe1365978a282178/sys/amd64/vmm/amd/vmcb.h#L195 > > https://github.com/freebsd/freebsd-src/blob/4c3a868d13c053ef173268cdfe1365978a282178/sys/amd64/vmm/amd/vmcb.h#L128C1-L128C30 > > XSETBV > > https://github.com/freebsd/freebsd-src/blob/4c3a868d13c053ef173268cdfe1365978a282178/sys/amd64/vmm/amd/vmcb.h#L94 > > I also asked Proton's Lumo+ AI what is 0x60 to see what it comes up with, > it said the following: > > exit_reason[0] = 0x60 is the VM‑exit code that bhyve reports when the > guest triggers the XSETBV instruction. > Why it happens > > XSETBV writes to the extended control register XCR0. > In Intel VMX (the virtualization technology bhyve uses on x86), > execution of XSETBV is not allowed to run directly in the guest; the CPU > forces a VM‑exit so the hypervisor can validate the operation. > bhyve records the exit reason as 0x60, which maps to the VMX exit > reason “XSETBV” in the Intel specification. > > What it means for you > > The guest tried to enable or change AVX‑512 / XSAVE feature bits via > XSETBV. > Because bhyve does not currently emulate that instruction, it aborts > the guest’s instruction stream and reports the exit code. > In most cases the guest will continue after bhyve handles the exit, > but if the hypervisor isn’t set up to emulate XSETBV you’ll see the VM stop > or crash. > > Typical ways to deal with it > > Update bhyve – newer releases include basic XSETBV handling for common > feature sets. > Adjust the guest’s CPU flags – start the VM with a CPU model that > doesn’t expose the XSAVE/XGETBV features the guest expects (e.g., -c host > vs. a specific model). > Patch bhyve – if you’re comfortable with C, you can add a handler for > VMX_EXIT_REASON_XSETBV that either emulates the instruction or simply > ignores it when the requested bits are already enabled. > > If you keep seeing the same exit repeatedly, it usually points to a > mismatch between the guest’s expectations (often a modern Linux kernel or a > Windows build that wants AVX‑512) and the capabilities bhyve is presenting. > Aligning those expectations—either by tweaking the VM’s CPU configuration > or by upgrading bhyve—generally resolves the issue. > > > Jonathan Vasquez > PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 > Sent with ProtonMail Secure Email > > > On Saturday, September 20th, 2025 at 19:47, Nuno Teixeira < > eduardo@freebsd.org> wrote: > > Hello Jonathan! > > Thanks for excellent quality videos on yt, I already subscribed it :) > > Just a quick questions: > > 1- Are you using freerdp3 to connect to win11pro? (Asking this because vnc > is very slow) > 2- Do you know how is nvidia status compared to your use case? > > Thanks! > > Jonathan Vasquez <jon@xyinn.org> escreveu (sábado, 20/09/2025 à(s) 02:42): > >> Hey all, I've spent some time today doing a gaming demo. I've uploaded >> the video in 2K so you can get the maximum effect. It's been an interesting >> experience so far, and I'm going to try to make it my primary gaming >> machine. We'll see how it goes. It's definitely qwirky, and sometimes I >> need to reboot the VM 4-5 times before it "stabilizes" (which means the VM >> won't just randomly crash or have massive lag in game). I'll need to >> collect more data through play testing. >> >> You can take a look at the video here (I'll post timestamps in the video >> soon): >> >> https://youtu.be/_cz0RUAw5p8 >> >> Jonathan Vasquez >> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 >> Sent with ProtonMail Secure Email >> >> >> >> On Wednesday, September 17th, 2025 at 22:33, Jonathan Vasquez < >> jon@xyinn.org> wrote: >> >> > Alright! The YouTube Video is up. Let me know what you think. I'm >> really excited about this. >> > >> > https://youtu.be/Ob4-v7dTJGs >> >> > > -- > Nuno Teixeira > FreeBSD UNIX: <eduardo@FreeBSD.org> Web: https://FreeBSD.org > <https://freebsd.org/> > > > > > >