Host filesystem corruption after bhyve error (bug report help)

[Nyakov Nyakovski]

Hi.

FreeBSD 12.2
I getting host filesystem (UFS and ZFS) corruption after bhyve error.
I try to passthru Chelsio T540-CR network card with SR-IOV to bhyve vm on PCI-E 2.0 slot.
ASUS F2A85M motherboard (A85X chipset) A10-5800K CPU.


I got bhyve error:
"bhyve: PCI device at 1/1/3 is not using the ppt(4) driverdevice emulation initialization error: No such file or directory"

And if after this I try to edit /etc/rc.conf file(for example), then reboot(system can refuse to turn off in the end of shutdown process), I will get corrupted root filesystem.
It happens on UFS and ZFS.

A was provided with clue that bhyve error emerging because PCIe slot does not support ARI.

But file corruption is whole other matter. I want to make bug report but have not enough knowledge to debug this issue or even determine with subsystem is causing it.

I need help with some debugging perhaps, bugreport guidance and any additional info that someone can provide.
Perhaps it is known issue?

Bits of my config:

/boot/loader.conf:

hw.vmm.amdvi.enable=1
vmm_load="YES"
nmdm_load="YES"
t5fw_cfg_load="YES"
if_cxgbe_load="YES"


/etc/rc.conf:

iovctl_enable="YES"
iovctl_files="/etc/iovctl/cxl3.conf"


/etc/iovctl/cxl3.conf:

PF {
    device : "cxl3";
    num_vfs : 2;
}
DEFAULT {
	passthrough : true;
}
VF-0 {passthrough : true;}
VF-1 {passthrough : true;}


#pciconf -l

t5iov0 at pci0:1:0:0:	class=0x020000 card=0x00001425 chip=0x50031425 rev=0x00 hdr=0x00
t5iov1 at pci0:1:0:1:	class=0x020000 card=0x00001425 chip=0x50031425 rev=0x00 hdr=0x00
t5iov2 at pci0:1:0:2:	class=0x020000 card=0x00001425 chip=0x50031425 rev=0x00 hdr=0x00
t5iov3 at pci0:1:0:3:	class=0x020000 card=0x00001425 chip=0x50031425 rev=0x00 hdr=0x00
t5nex0 at pci0:1:0:4:	class=0x020000 card=0x00001425 chip=0x54031425 rev=0x00 hdr=0x00
none2 at pci0:1:0:5:	class=0x010000 card=0x00001425 chip=0x55031425 rev=0x00 hdr=0x00
none3 at pci0:1:0:6:	class=0x0c0400 card=0x00001425 chip=0x56031425 rev=0x00 hdr=0x00
vgapci1 at pci0:3:0:0:	class=0x030200 card=0x377b1458 chip=0x1c0710de rev=0xa1 hdr=0x00
ppt0 at pci0:1:1:3:	class=0xffffff card=0x00000000 chip=0x58031425 rev=0xff hdr=0x7f
ppt1 at pci0:1:1:7:	class=0xffffff card=0x00000000 chip=0x58031425 rev=0xff hdr=0x7f

Last two is devices created from /etc/iovctl/cxl3.conf


bhyve command:

bhyve -A -H -S -P -w \
                -s 0:0,amd_hostbridge \
                -s 1:0,passthru,1/1/3 \
                -s 4:0,ahci-cd,"artix-lxqt-openrc-20201128-x86_64.iso" \
                -s 29,fbuf,tcp="10.35.1.2:2911",w=1024,h=768,wait \
                -s 30,xhci,tablet \
                -s 31:0,lpc \
                -c sockets=1,cores=1,threads=1 \
                -m 2G \
                -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
                artix-linux-nvidia-passthrough


Bits from dmesg:

t5nex0: <Chelsio T540-CR> mem 0xfc300000-0xfc37ffff,0xfb000000-0xfbffffff,0xfc984000-0xfc985fff irq 16 at device 0.4 on pci1
t5nex0: Disabled No Snoop/Relaxed Ordering on pcib1
cxl0: <port 0> on t5nex0
cxl0: Ethernet address: xx:xx:xx:xx:xx:xx
cxl0: 4 txq, 4 rxq (NIC); 4 txq (TOE), 2 rxq (TOE)
cxl1: <port 1> on t5nex0
cxl1: Ethernet address: xx:xx:xx:xx:xx:xx
cxl1: 4 txq, 4 rxq (NIC); 4 txq (TOE), 2 rxq (TOE)
cxl2: <port 2> on t5nex0
cxl2: Ethernet address: xx:xx:xx:xx:xx:xx
cxl2: 4 txq, 4 rxq (NIC); 4 txq (TOE), 2 rxq (TOE)
cxl3: <port 3> on t5nex0
cxl3: Ethernet address: xx:xx:xx:xx:xx:xx
cxl3: 4 txq, 4 rxq (NIC); 4 txq (TOE), 2 rxq (TOE)
t5nex0: PCIe gen2 x4, 4 ports, 26 MSI-X interrupts, 60 eq, 25 iq

ivhd0: <AMD-Vi/IOMMU ivhd> on acpi0
ivhd0: Flag:fe<PassPW,ResPassPW,Isoc,IotlbSup,Coherent,PreFSup>
ivhd0: Features(type:0x10) HATS = 0 GATS = 0 MsiNumPPR = 0 PNBanks= 2 PNCounters= 4
ivhd0: max PASID = 8 GLXSup = 0 Feature:48824<GTSup,GASup>
ivhd0: supported paging level:7, will use only: 4
ivhd0: device range: 0x0 - 0xfffe
ivhd0: PCI cap 0x90b540f at 0x40 feature:9<IOTLB,EFR>