Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro)

I've also started to upload some of these crash files (like the 0x88.0.txt file) to Lumo for analysis. The AI info is interesting but it tends to have errors on invalid bhyve command line flags or sometimes even what code it thinks 0x88 belongs to. I'll post some of it below. This was its response to the 0x88.0.txt data. If we look at what it says for the 0x88 code, it doesn't match what I see in the freebsd code (#define	VMCB_EXIT_ICEBP			0x88): https://github.com/freebsd/freebsd-src/blob/4c3a868d13c053ef173268cdfe1365978a282178/sys/amd64/vmm/amd/vmcb.h#L94

is the AI wrong about this?

---- Lumo Response:

I can see you’ve uploaded a text dump named 0x88.0.txt that looks like a snapshot of a bhyve/AMD‑SVM virtual‑machine (VM) state and statistics. Here’s a concise walkthrough of what the file contains and what it tells us about the VM’s execution.

1. High‑level picture

Item	Value / Meaning
VM exit reason	0x88 – this is the exit code for VM_EXIT_REP_INSB (a repeated string‑input operation, i.e., rep insb/insw/insd).
Current exception vector	6 – a #UD (invalid opcode) or #GP depending on context; the dump shows the VM is presently handling an exception.
Number of CPUs	16 vCPUs (IDs 0‑15) are active and suspended; the host has a 16‑core topology.
Total VM exits	3 259 037 – a fairly large number, indicating heavy interaction between guest and hypervisor.
Instruction‑emulation exits	1 465 913 – most of the exits are for instructions that bhyve had to emulate (e.g., I/O, privileged ops).
Pause‑instruction exits	1 138 635 – the guest frequently executed pause, which bhyve intercepts for power‑saving.
HLT exits	75 539 – the guest entered the halt state many times (often seen in idle loops).
In/out exits	492 316 – the guest performed port‑I/O operations that required hypervisor handling.
CPUID exits	3 735 – the guest queried CPU features.
MSR reads/writes	1 783 reads, 95 writes – typical for a modern OS that accesses model‑specific registers.
Nested page‑fault exits	610 – indicates the guest’s second‑level address translation (NPT/EPT) missed some pages.
Memory layout	Two huge sysmem regions (≈33 GB and ≈29 GB) plus a 16 KB boot ROM region. Most of the RAM is “wired” (pinned) for the VM.
RTC time	Mon Sep 22 10:35:00 2025 – matches today’s date, confirming the snapshot is recent.


2. What the 0x88 exit really means

    Opcode involved: rep insb/insw/insd (string input from an I/O port).
    Why bhyve exits: AMD‑SVM (and Intel VT‑x) cannot directly execute this instruction in guest mode; they force a VM exit so the hypervisor can either emulate the I/O or inject a #DB (debug) exception.
    Typical causes:
        Guest reading from a legacy serial port, keyboard controller, or other device that uses bulk input.
        Anti‑debugging tricks where malware repeatedly executes rep insb to detect a hypervisor.
    Impact: Each such exit costs a few microseconds of latency. A high frequency can degrade performance, especially if the guest is in a tight loop doing bulk reads.


3. Other notable fields
Field	Interpretation
cr_intercept = 0xfee2fee2	The hypervisor is set to intercept many control‑register accesses (CR0, CR3, CR4, etc.).
exc_intercept = 0x00040000	Exception interception is enabled for the #PF (page‑fault) vector.
inst1_intercept = 0xddc4000f	Intercepts a specific instruction pattern (likely hlt and pause).
exception_bitmap = 0x40000	Only the page‑fault exception (bit 18) is being intercepted.
io_bitmap = 0x2b053b000	Defines which I/O ports cause exits; the presence of many in/out exits suggests a fairly wide bitmap.
msr_bitmap = 0x2b0539000	Controls which Model‑Specific Register accesses trigger exits.
capability flags	The VM supports hlt_exit, pause_exit, unrestricted_guest, ipi_exit; lacks mtrap_exit, enable_invpcid, etc.
guest_pat = 0x0007010600070106	PAT (Page Attribute Table) settings – standard write‑combining / write‑through configuration.
asid = 0x07fd	Address‑Space Identifier used for NPT/EPT – helps reduce TLB flushes.
rflags = 0x40193	Typical user‑mode flag state (IF=1, ZF=0, etc.).
rip = 0xffffd001b2ce151d	Current instruction pointer; the address falls within the first sysmem region, suggesting the guest is executing normal code when the dump was taken.
rsp = 0xffffe3018da8e7b8	Stack pointer – also in the wired sysmem area.


4. Performance clues

    High pause‑exit count (>1 M) + high hlt‑exit count (~75 k) indicate the guest spends a lot of time in idle loops. That’s typical for a server VM that sleeps when idle.
    Large number of instruction‑emulation exits (~1.46 M) suggests the guest is doing a fair amount of privileged I/O (e.g., disk, network, or device drivers) that the hypervisor must emulate.
    Relatively low NMI and ExtINT counts (0) imply the VM isn’t receiving asynchronous hardware interrupts, which aligns with a mostly self‑contained workload.


5. What you might want to investigate next

    Why so many rep insb exits?
        Look at the guest’s I/O port map (e.g., serial console, legacy floppy). If the guest is repeatedly reading from a port that isn’t needed, you could disable that device in the bhyve config.
        If the exits are caused by malicious anti‑debug code, consider adding a pass‑through for that port or using a more permissive I/O bitmap.

    Tune the I/O bitmap
        If you know which ports the guest actually needs, narrow the bitmap to reduce unnecessary exits.

    Check pause‑exit handling
        Modern kernels already use pause for spin‑waits; the high count is normal, but you can enable the pause_exit capability (already set) to let the hypervisor handle them efficiently.

    Memory wiring
        Almost all RAM is marked “wired/iommu”. If you don’t need that much pinned memory, you could relax the wiring to allow the host to reclaim pages under pressure.

    Look at the instruction pointer (rip)
        Mapping 0xffffd001b2ce151d back to the guest’s binary (via objdump/gdb on the guest image) could reveal exactly which instruction caused the 0x88 exit. That can confirm whether it’s a legitimate driver read or something suspicious.

TL;DR

Your dump shows a bhyve VM that is currently handling a VM_EXIT_REP_INSB (0x88) – a bulk input operation from an I/O port. The VM has been quite busy: over three million exits, many of which are for instruction emulation, pause, and I/O. Memory is largely wired, and the guest is idle‑heavy (lots of hlt/pause). If you want to improve performance or diagnose a possible anti‑debugging pattern, focus on the ports generating the rep insb exits and consider tightening the I/O bitmap or providing a proper device backend.