bhyve + USB disks as slots + virtio-blk driver + passt-thru will not allow any OS to boot

Hello.

What I want to achieve is to pass thru two of my NTFS "formatted" disks to
a Windows 11 VM,but without passing them thru using the USB controller in
FreeBSD with a bhyve virtual machine (in the example below I tried to boot
Windows 11 from the nvme disk nvd0) AT at the same time I want to pass thru
my graphic card to a Windows 11 and / or Linux VM.

I'm using this FreeBSD version :

*FreeBSD marietto 13.0-RELEASE FreeBSD 13.0-RELEASE #5
n244809-dff3dead3734: Wed Feb 23 13:16:32 CET
2022     marietto@marietto:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64*


I've configured the bhyve VM like this :

*bhyve -S -c sockets=1,cores=2,threads=2 -m 4G -w -H -A \
-s 0,hostbridge \
-s 1,ahci-hd,/dev/nvd0 \
-s 2,virtio-blk,/dev/da4 \
-s 3,virtio-blk,/dev/da2 \
-s 4:0,passthru,2/0/0 \
-s 4:1,passthru,2/0/1 \
-s 4:2,passthru,2/0/2 \
-s 4:3,passthru,2/0/3 \
-s 8,virtio-net,tap2 \
-s 9,virtio-9p,sharename=/ \
-s 10,hda,play=/dev/dsp,rec=/dev/dsp \
-s 29,fbuf,tcp=0.0.0.0:5902 <http://0.0.0.0:5902>,w=1440,h=900 \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \
vm2 < /dev/null & sleep 2 && vncviewer 0:2*


These are the NTFS disks that I would like to see inside the Windows 11
guest os :

-s 2,virtio-blk,/dev/da4 \
-s 3,virtio-blk,/dev/da2 \
*
=>         34  19532873661  da4  GPT  (9.1T)
           34        32734    1  ms-reserved  (16M)
32768  19532838912    2  ms-basic-data  (9.1T)
19532871680         2015       - free -  (1.0M)


=>         34  23437705149  da2  GPT  (11T)
           34         2014       - free -  (1.0M)
2048  23437701120    1  ms-basic-data  (11T)
23437703168         2015       - free -  (1.0M)*


Using the whole disks in bhyve they are recognized by fdisk. But as I
said,what matters a lot for me,is to boot the nvme physical disk passing
through at the same time my graphic card but it does not work. I care so
much about this feature,because Ubuntu boots much much faster if I use the
physical disk instead of the img / raw file. And this is again more valid
if I want to boot Windows 11. It gets frozen e if I use a raw / img disk. I
can't use it. But if I use the nvme disk where I have installed it
physically it is much,much faster and I can use it.

This is what happens : (linux and windows 11 installed on the disk nvme
don't boot)

https://forums.freebsd.org/attachments/screenshot_2022-03-13_16-35-42-jpg.13323/

I will keep the discussion updated also here :

*https://forums.freebsd.org/threads/usb-3-0-disks-not-recognized-by-windows-if-passed-through-as-slots.84402/#post-559892
<https://forums.freebsd.org/threads/usb-3-0-disks-not-recognized-by-windows-if-passed-through-as-slots.84402/#post-559892>*


-- 
Mario.