No physically installed OS can boot if a device is passed through at the same time,using any kind of recent FreeBSD version as host OS (13R,13-STABLE and 14-CURRENT are involved)

Hello to everyone.

I'm using bhyve on FreeBSD 13R p7 and I'm trying to pass thru my USB
controller and / or my graphic card (Geforce RTX 2080 ti) while at the same
time I would like to boot a physically installed OS. I can choose between 3
operating systems (Ubuntu,Windows and FreeBSD 13) that I have installed
physically on a SATA and / or NVME disk and I'm trying to call them inside
the bhyve parameter using the virtio-blk driver and the ahci-hd driver. The
problems that come out from this setup are different and concatenated.
Anyway,those OS won't boot because it seems that the devices that I pass
thru interferes with the booting process. If I don't pass thru any
device,they can boot properly. In addition,my graphic card seems to have an
additional problem. It seems that it is not passed correctly when I use
FreeBSD 13R (it is if I use FreeBSD 14 Current and 13-stable). Below I want
to show you different cases to explain the problems that I'm experimenting :

CASE 0) Without passing thru the USB controller and the graphic card on
Ubuntu 21.10 / Windows 11 (installed physically) can boot properly with
these parameters On FreeBSD 13R :

bhyve -S -c sockets=1,cores=2,threads=2 -m 4G -w -H \
-s 0,hostbridge \
-s 1,ahci-hd,/dev/nvd0 \
-s 8,virtio-net,tap1 \
-s 29,fbuf,tcp=0.0.0.0:5901,w=1440,h=900,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
-l com1,stdio \
vm1

BdsDxe: loading Boot0001 "UEFI BHYVE SATA DISK BHYVE-51F3-061F-A385" from
PciRoot(0x0)/Pci(0x1,0x0)/Sata(0x0,0xFFFF,0x0)
BdsDxe: starting Boot0001 "UEFI BHYVE SATA DISK BHYVE-51F3-061F-A385" from
PciRoot(0x0)/Pci(0x1,0x0)/Sata(0x0,0xFFFF,0x0)

CASE 1) Passing thru the USB controller and the graphic card and booting
from a physical installation of Ubuntu 21.10 / Windows 11 (and also with
the physical installation of FreeBSD 14 CURRENT as guest os) on a nvme disk
on FreeBSD 13R :

bhyve -S -c sockets=1,cores=2,threads=2 -m 4G -w -H \
-s 0,hostbridge \
-s 1,ahci-hd,/dev/nvd0 \
-s 2,passthru,1/0/0 \
-s 3:0,passthru,2/0/0 \
-s 3:1,passthru,2/0/1 \
-s 3:2,passthru,2/0/2 \
-s 3:3,passthru,2/0/3 \
-s 8,virtio-net,tap1 \
-s 29,fbuf,tcp=0.0.0.0:5901,w=1440,h=900,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
-l com1,stdio \
vm1

Sometimes Assertion failed: (error == 0), function modify_bar_registration,
file /usr/src/usr.sbin/bhyve/pci_emul.c, line 501: Abort / sometimes black
screen frozen

CASE 2) Passing thru the USB controller and booting from a physical
installation of Ubuntu 21.10 / Windows 11 on a nvme disk (and also with the
physical installation of FreeBSD 14 CURRENT as guest os) on FreeBSD 13R :

bhyve -S -c sockets=1,cores=2,threads=2 -m 4G -w -H \
-s 0,hostbridge \
-s 1,ahci-hd,/dev/nvd0 \
-s 2,passthru,1/0/0 \
-s 8,virtio-net,tap1 \
-s 29,fbuf,tcp=0.0.0.0:5901,w=1440,h=900,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
-l com1,stdio \
vm1

= BdsDxe: failed to load Boot0001 "UEFI BHYVE SATA DISK
BHYVE-51F3-061F-A385" from PciRoot(0x0)/Pci(0x1,0x0)/Sata(0x0,0xFFFF,0x0):
Not Found

CASE 3) Passing thru the graphic card and booting from a physical
installation of Ubuntu 21.10 / Windows 11 on a nvme disk (and also with the
physical installation of FreeBSD 14 CURRENT as guest os) on FreeBSD 13R :

bhyve -S -c sockets=1,cores=2,threads=2 -m 4G -w -H \
-s 0,hostbridge \
-s 1,ahci-hd,/dev/nvd0 \
-s 2:0,passthru,2/0/0 \
-s 2:1,passthru,2/0/1 \
-s 2:2,passthru,2/0/2 \
-s 2:3,passthru,2/0/3 \
-s 8,virtio-net,tap1 \
-s 29,fbuf,tcp=0.0.0.0:5901,w=1440,h=900,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
-l com1,stdio \
vm1

= Assertion failed: (error == 0), function modify_bar_registration, file
/usr/src/usr.sbin/bhyve/pci_emul.c, line 501 : Abort

case 4) Booting from a physical installation of Ubuntu 21.10 / Windows 11
on a nvme disk (and also with the physical installation of FreeBSD 13R as
guest os) on FreeBSD 13-STABLE :

bhyve -S -c sockets=1,cores=2,threads=2 -m 4G -w -H \
-s 0,hostbridge \
-s 1,ahci-hd,/dev/nvd0 \
-s 8,virtio-net,tap1 \
-s 29,fbuf,tcp=0.0.0.0:5901,w=1440,h=900,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
-l com1,stdio \
vm1

it can boot.

case 5) Passing thru the graphic card and booting from a physical
installation of Ubuntu 21.10 / Windows 11 on a nvme disk (and also with the
physical installation of FreeBSD 13R as guest os) on FreeBSD 13-STABLE :

bhyve -S -c sockets=1,cores=2,threads=2 -m 4G -w -H \
-s 0,hostbridge \
-s 1,ahci-hd,/dev/nvd0 \
-s 2:0,passthru,2/0/0 \
-s 2:1,passthru,2/0/1 \
-s 2:2,passthru,2/0/2 \
-s 2:3,passthru,2/0/3 \
-s 8,virtio-net,tap1 \
-s 29,fbuf,tcp=0.0.0.0:5901,w=1440,h=900,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
-l com1,stdio \
vm1

it won't boot : BdsDxe: failed to load Boot0001 "UEFI BHYVE SATA DISK
BHYVE-EE1E-05D9-14BB" from PciRoot(0x0)/Pci(0x1,0x0)/Sata(0x0,0xFFFF,0x0):
Not Found

case 6) Passing thru my USB controller and the graphic card and booting
from a physical installation of Ubuntu 21.10 / Windows 11 on a nvme disk
(and also with the physical installation of FreeBSD 13R as guest os) on
FreeBSD 13-STABLE :

bhyve -S -c sockets=1,cores=2,threads=2 -m 4G -w -H \
-s 0,hostbridge \
-s 1,ahci-hd,/dev/nvd0 \
-s 2:0,passthru,1/0/0 \
-s 3:0,passthru,2/0/0 \
-s 3:1,passthru,2/0/1 \
-s 3:2,passthru,2/0/2 \
-s 3:3,passthru,2/0/3 \
-s 8,virtio-net,tap1 \
-s 29,fbuf,tcp=0.0.0.0:5901,w=1440,h=900,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
-l com1,stdio \
vm1

it won't boot : BdsDxe: failed to load Boot0001 "UEFI BHYVE SATA DISK
BHYVE-EE1E-05D9-14BB" from PciRoot(0x0)/Pci(0x1,0x0)/Sata(0x0,0xFFFF,0x0):
Not Found.

SUMMARY : It seems there is some serious bug in every version of freebsd,13
release,13 stable and 14 current,that does not allow to boot any kind of
physically installed OS if it is passed through any kind of device. I've
already opened a bug report and no one seems to work on it. Would someone
take care of these bugs ? thanks.

-- 
Mario.