[Bug 290155] bhyve PCI passthrough regression in 14.3-RELEASE: VMM IOMMU fails to initialize on Intel systems

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=290155

            Bug ID: 290155
           Summary: bhyve PCI passthrough regression in 14.3-RELEASE: VMM
                    IOMMU fails to initialize on Intel systems
           Product: Base System
           Version: 14.3-RELEASE
          Hardware: amd64
                OS: Any
            Status: New
          Severity: Affects Some People
          Priority: ---
         Component: bhyve
          Assignee: virtualization@FreeBSD.org
          Reporter: luc+freebsd@2113.ch

Created attachment 264491
  --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=264491&action=edit
dmesg output on workin host 14.2

Host Kernel: 14.2-RELEASE-p1
FreeBSD app-01 14.2-RELEASE-p1 FreeBSD 14.2-RELEASE-p1 GENERIC amd64

=== HARDWARE ===
hostb0@pci0:0:0:0:      class=0x060000 rev=0x0d hdr=0x00 vendor=0x8086
device=0x3e30 subvendor=0x1043 subdevice=0x8694
    vendor     = 'Intel Corporation'
    device     = '8th/9th Gen Core 8-core Desktop Processor Host Bridge/DRAM
Registers [Coffee Lake S]'
    class      = bridge

CPU: Intel Coffee Lake (8th/9th Gen Core)
Chipset: Intel Z370
Motherboard: ASUS (VT-d enabled in BIOS)

=== CONFIGURATION ===
Loader Configuration (/boot/loader.conf):
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
cryptodev_load="YES"
zfs_load="YES"


#
# bhyve
#

vmm_load="YES"
nmdm_load="YES"
if_bridge_load="YES"
if_tap_load="YES"

# if_cxgbe_load="YES"

# Mellanox ConnectX-4 Lx
# mlx5_core0@pci0:1:0:0:  class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3
device=0x1015 subvendor=0x1590 subdevice=0x00d4
# mlx5_core0@pci0:1:0:1:  class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3
device=0x1015 subvendor=0x1590 subdevice=0x00d4


# Samsung 990 PRO
# nvme3@pci0:7:0:0:       class=0x010802 rev=0x00 hdr=0x00 vendor=0x144d
device=0xa80c subvendor=0x144d subdevice=0xa801
#     vendor     = 'Samsung Electronics Co Ltd'
#     device     = 'NVMe SSD Controller S4LV008[Pascal]'
#     class      = mass storage
#     subclass   = NVM

# Add to existing pptdevs or create new line

pptdevs="1/0/0 7/0/0"



Devices for Passthrough:
- Mellanox ConnectX-4 Lx (dual-port 10/25GbE): PCI 1:0:0 and 1:0:1
- Samsung NVMe SSD: PCI 7:0:0

=== WORKING STATE (14.2-RELEASE-p7 as host) ===
VT-d Status:
  DMAR: Length=176, Revision=1, Checksum=255,

DMAR Detection:

PPT Device Attachment:
ppt0@pci0:1:0:0:        class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3
device=0x1015 subvendor=0x1590 subdevice=0x00d4
ppt1@pci0:7:0:0:        class=0x010802 rev=0x00 hdr=0x00 vendor=0x144d
device=0xa80c subvendor=0x144d subdevice=0xa801

ppt0 mem 0x2ffc000000-0x2ffdffffff irq 16 at device 0.0 on pci1
ppt1 mem 0xdfa00000-0xdfa03fff irq 16 at device 0.0 on pci7

VMM Status:
hw.vmm.iommu.enable: 1
hw.vmm.iommu.initialized: 1
hw.vmm.ppt.devices: 2

VM Status:
store-01         default    bhyveload  8    16384M  -    Yes [1]  Running
(37745)

Result: VM boots successfully, devices are accessible inside guest

Guest OS pciconf output (showing passed-through devices):
- mlx5_core0@pci0:0:5:0: Mellanox ConnectX-4 Lx (working)
- nvme2@pci0:0:6:0: Samsung NVMe SSD (working)
Guest is running FreeBSD 14.3-RELEASE-p4 successfully.

=== BROKEN STATE (14.3-RELEASE-p3/p4 as host) ===

Configuration: Identical to 14.2 (same loader.conf, same hardware)

VT-d Status: ENABLED (same DMAR table present)

DMAR Detection: 
- dmar0 and dmar1 detected
- Regular devices being re-mapped successfully
- DMAR is functional

PPT Device Attachment:
- ppt0, ppt1, ppt2 show in pciconf
- dmesg shows successful hardware attachment
- hw.vmm.ppt.devices = 2 (VMM sees devices)

VMM Status:
- hw.vmm.iommu.enable = 1
- hw.vmm.iommu.initialized = 0  _ CRITICAL FAILURE
- hw.vmm.vmx.initialized = 1 (Intel VMX working)

Result: 
- VM crashes immediately when started with passthrough devices
- Boot gets to kernel load, then crashes during hardware initialization
- Exit code 4 (reboot/crash)

Workaround Attempts on 14.3:
1. Setting hw.vmm.iommu.enable="0" in loader.conf:
   - Result: hw.vmm.ppt.devices becomes 0, devices not detected at all

2. Setting hw.vmm.iommu.enable="1" in loader.conf:
   - Result: hw.vmm.iommu.initialized stays 0, VM crashes

3. Setting hw.dmar.enable="1" in loader.conf:
   - Result: No change, IOMMU still not initialized

No configuration allows VMM IOMMU to initialize on 14.3-RELEASE.

=== ROOT CAUSE ANALYSIS ===

The AMD IOMMU driver integration work that was merged into stable/14 
for 14.3-RELEASE appears to have introduced a regression affecting 
Intel DMAR system ?

Key observations:
1. DMAR hardware is functional (regular device remapping works)
2. ppt driver attaches successfully to hardware
3. VMM recognizes ppt devices
4. BUT: VMM IOMMU subsystem fails to initialize
5. Without VMM IOMMU init, bhyve cannot safely use passthrough
6. Result: VM crashes when passthrough devices are configured

This appears to be a change in initialization order or requirements 
where 14.3 now requires VMM IOMMU to be initialized before allowing 
passthrough, but the Intel DMAR integration with VMM IOMMU is broken.

References from FreeBSD Status Reports:
- Q2 2024: "AMD IOMMU driver... will include bhyve integration"
- Q4 2024: "AMD IOMMU driver... was pushed to src tree in early November"
- The driver is in stable/14 branch (14.3-RELEASE)

=== REGRESSION CONFIRMATION ===

Tested using FreeBSD boot environments:
- Host: 14.2-RELEASE-p7 (Oct 11, 2025) = WORKS
- Host: 14.3-RELEASE-p3 (Oct 11, 2025) = FAILS


Same hardware, same BIOS settings, same loader.conf configuration.
The only variable is the FreeBSD version on the HOST system.

Reproducibility: 100% reproducible on Intel Z370 chipset

=== IMPACT ===

CRITICAL for users upgrading from 14.2 to 14.3:
- All bhyve PCI passthrough configurations will break on Intel systems
- Silent failure: appears to configure correctly but VMs crash at boot
- No error messages indicating IOMMU initialization failure
- No workaround exists within 14.3-RELEASE

Affected users:
- Anyone using bhyve with PCI passthrough on Intel systems
- GPU passthrough configurations
- Network card passthrough (SR-IOV, dedicated NICs)
- NVMe drive passthrough
- Any PCI device passthrough scenario

=== EXPECTED BEHAVIOR ===

When VT-d is enabled and DMAR is functional:
1. hw.vmm.iommu.initialized should equal 1
2. PCI passthrough should work as it does in 14.2-RELEASE
3. OR: System should gracefully degrade to 14.2 behavior if IOMMU init fails

=== ACTUAL BEHAVIOR ===

On 14.3-RELEASE with Intel DMAR:
1. hw.vmm.iommu.initialized always equals 0
2. VM crashes when passthrough devices are configured
3. No error messages explaining the failure
4. No graceful fallback to non-IOMMU passthrough mode

=== WORKAROUND ===

Stay on FreeBSD 14.2-RELEASE as host OS.
Boot environments make this easy to maintain while waiting for fix.

=== RECOMMENDED FIX ===

1. Debug why VMM IOMMU initialization fails on Intel DMAR systems
2. Fix initialization to properly detect and use Intel DMAR
3. OR: Allow passthrough to work without strict IOMMU init (14.2 behavior)
4. Add error messages when IOMMU init fails to aid debugging

=== ADDITIONAL NOTES ===

- Guest OS version doesn't matter (can run 14.3 guest on 14.2 host)
- Only affects 14.3-RELEASE as HOST, not as guest
- Problem is in VMM kernel module's IOMMU integration
- May affect other Intel chipsets beyond Z370

=== FILES ATTACHED ===

1. Full dmesg from 14.2-RELEASE-p7 (working)
2. Full sysctl hw.vmm output from 14.2
3. pciconf -lv output showing device configuration
4. Guest pciconf output showing successful passthrough


=== Bug report created at /tmp/freebsd-ppt-bug-14.3.txt ===
=== Supporting files:
    /tmp/dmesg-14.2-working.txt
    /tmp/sysctl-vmm-14.2.txt
    /tmp/pciconf-14.2.txt

-- 
You are receiving this mail because:
You are the assignee for the bug.