[Bug 241774] FreeBSD 11.3 & 12.0 has broken SCSI & Networking on KVM/QEMU Q35 with OVMF
bugzilla-noreply at freebsd.org
bugzilla-noreply at freebsd.org
Sun Jan 12 02:22:39 UTC 2020
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241774
--- Comment #29 from Tommy P <tommyhp2 at gmail.com> ---
John & Matt,
What is/are the CPU/s for your host OS? Did you verify that the relevant
virtual technology features are enabled on the host: IOMMU for AMD and VT/D (?)
for Intel. IIRC, some older versions of Linux (CentOS, Ubuntu, and Fedora)
requires additional manual entry in the grub's boot loader. The modern Linux
_shouldn't_ but doesn't hurt to confirm:
dmesg|grep -i ommu
[ 0.000000] AGP: Please enable the IOMMU option in the BIOS setup
[ 3.434650] iommu: Adding device 0000:00:00.0 to group 0
<snip/>
[ 3.536829] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
[ 3.536834] AMD-Vi: Found IOMMU at 0000:40:00.2 cap 0x40
grep -e '^#' -v /etc/default/grub
GRUB_DEFAULT=0
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX=""
GRUB_DISABLE_SUBMENU=y
How are you creating/managing your VMs on the host? I use virt-manager (to
reduce typos, misconfiguration, or unsupported configuation errors) on Ubuntu
to manage. Did you customize any of the configurations for the VM in the shell
script (if used to maintain the VMs) or XML backend?
Here's a screen shot for the VM's config: https://imgur.com/a/1WbzvhQ
Below is the command line startup for that VM without any manual customization
in the XML configuration file:
ps -ef|grep 'fbsd11_ovmf'
root 8914 20540 0 15:19 pts/7 00:00:00 grep --color=auto fbsd11_ovmf
libvirt+ 25561 1 40 Jan09 ? 17:21:30 qemu-system-x86_64 -enable-kvm
-name guest=z_fbsd12_ovmf,debug-threads=on -S
-object
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-35-z_fbsd12_ovmf/master-key.aes
-machine pc-q35-2.11,accel=kvm,usb=off,vmport=off,dump-guest-core=off
-cpu
Opteron_G5,vme=on,ht=on,monitor=on,osxsave=on,bmi1=on,mmxext=on,fxsr_opt=on,cmp_legacy=on,extapic=on,cr8legacy=on,osvw=on,ibs=on,skinit=on,wdt=on,tce=on,nodeid_msr=on,topoext=on,perfctr_core=on,perfctr_nb=on,invtsc=on,ibpb=on
-drive
file=/usr/share/OVMF/OVMF_CODE.fd,if=pflash,format=raw,unit=0,readonly=on
-drive
file=/var/lib/libvirt/qemu/nvram/z_fbsd12_ovmf_VARS.fd,if=pflash,format=raw,unit=1
<...snip...>
-no-user-config
-nodefaults
<...snip...>
-rtc base=utc,driftfix=slew
-global kvm-pit.lost_tick_policy=delay
-no-hpet
-no-shutdown
-global ICH9-LPC.disable_s3=1
-global ICH9-LPC.disable_s4=1
-boot menu=on,strict=on
-device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e
-device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x0 (<<< this should
be em0 NIC)
-device
pcie-root-port,port=0x10,chassis=3,id=pci.3,bus=pcie.0,multifunction=on,addr=0x2
(<<< this should be the VirtIO NIC)
-device pcie-root-port,port=0x11,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x1
(<<< sym0 LSI/Symbios SCSI)
-device pcie-root-port,port=0x12,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x2
(<<< VirtIO SCSI)
-device pcie-root-port,port=0x13,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x3
(<<< VirtIO Serial)
-device pcie-root-port,port=0x14,chassis=7,id=pci.7,bus=pcie.0,addr=0x2.0x4
(<<< VirtIO Memory Balloon)
-device pcie-root-port,port=0x15,chassis=8,id=pci.8,bus=pcie.0,addr=0x2.0x5
(<<< 'Controller PCIe 0' seen in the screen shot ?)
-device ich9-usb-ehci1,id=usb,bus=pcie.0,addr=0x1d.0x7
-device
ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pcie.0,multifunction=on,addr=0x1d
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pcie.0,addr=0x1d.0x1
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pcie.0,addr=0x1d.0x2
-device lsi,id=scsi0,bus=pci.4,addr=0x0 (<<< sym0 LSI/Symbios SCSI)
-device virtio-scsi-pci,id=scsi1,bus=pci.5,addr=0x0
-device virtio-serial-pci,id=virtio-serial0,bus=pci.6,addr=0x0
<...snip...>
-netdev tap,fd=39,id=hostnet0
-device e1000,netdev=hostnet0,id=net0,mac=52:54:00:4e:d4:8f,bus=pci.2,addr=0x1
-netdev tap,fd=41,id=hostnet1,vhost=on,vhostfd=42
-device
virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:a9:08:fb,bus=pci.3,addr=0x0
<...snip...>
-device
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0
<...snip...>
-device
qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pcie.0,addr=0x1
<...snip...>
-device virtio-balloon-pci,id=balloon0,bus=pci.7,addr=0x0
-msg timestamp=on
---------------------------------------------------------------
11.3-RELEASE-p5 r356613 boot log messages for em0 NIC:
root at fbsd11-ovmf:~ # egrep 'em0' /var/log/messages
Jan 10 00:29:26 fbsd11-ovmf kernel: em0: <Intel(R) PRO/1000 Legacy Network
Connection 1.1.0> port 0x6000-0x603f mem 0x98000000-0x9801ffff irq 21 at device
1.0 on pci2
Jan 10 00:29:26 fbsd11-ovmf kernel: em0: Ethernet address: 52:54:00:4e:d4:8f
Jan 10 00:29:26 fbsd11-ovmf kernel: em0: netmap queues/slots: TX 1/256, RX
1/256
Jan 10 00:29:26 fbsd11-ovmf kernel: em0: link state changed to UP
Jan 10 01:08:26 fbsd11-ovmf kernel: em0: <Intel(R) PRO/1000 Legacy Network
Connection 1.1.0> port 0x6000-0x603f mem 0x98000000-0x9801ffff irq 21 at device
1.0 on pci2
Jan 10 01:08:26 fbsd11-ovmf kernel: em0: Ethernet address: 52:54:00:4e:d4:8f
Jan 10 01:08:26 fbsd11-ovmf kernel: em0: netmap queues/slots: TX 1/256, RX
1/256
Jan 10 01:08:26 fbsd11-ovmf kernel: em0: link state changed to UP
Jan 10 01:26:48 fbsd11-ovmf kernel: em0: <Intel(R) PRO/1000 Legacy Network
Connection 1.1.0> port 0x6000-0x603f mem 0x98000000-0x9801ffff irq 21 at device
1.0 on pci2
Jan 10 01:26:48 fbsd11-ovmf kernel: em0: Ethernet address: 52:54:00:4e:d4:8f
Jan 10 01:26:48 fbsd11-ovmf kernel: em0: netmap queues/slots: TX 1/256, RX
1/256
Jan 10 01:26:48 fbsd11-ovmf kernel: em0: link state changed to UP
Jan 11 16:00:03 fbsd11-ovmf kernel: em0: <Intel(R) PRO/1000 Legacy Network
Connection 1.1.0> port 0x6000-0x603f mem 0x98000000-0x9801ffff irq 21 at device
1.0 on pci2
Jan 11 16:00:03 fbsd11-ovmf kernel: em0: Ethernet address: 52:54:00:4e:d4:8f
Jan 11 16:00:03 fbsd11-ovmf kernel: em0: netmap queues/slots: TX 1/256, RX
1/256
Jan 11 16:00:03 fbsd11-ovmf kernel: em0: link state changed to UP
---------------------------------------------------------------
Snippets of my custom kernel since 11.2-RELEASE-p15 and upgraded to
11.3-RELEASE-p5 r356613:
root at fbsd11-ovmf:~ # grep -e '^#' -v /usr/src/sys/amd64/conf/Custom|grep -e
'^$' -v
cpu HAMMER
ident GENERIC
options SCHED_ULE # ULE scheduler
options PREEMPTION # Enable kernel thread preemption
<snip>
device acpi
options ACPI_DMAR
device pci
options PCI_HP # PCI-Express native HotPlug
options PCI_IOV # PCI SR-IOV support
device ahci # AHCI-compatible SATA controllers
device ata # Legacy ATA/SATA controllers
<snip>
device em # Intel PRO/1000 Gigabit Ethernet
Family
device miibus # MII bus support
device re # RealTek 8139C+/8169/8169S/8110S
<snip>
device uhci # UHCI PCI->USB interface
device ohci # OHCI PCI->USB interface
device ehci # EHCI PCI->USB interface (USB 2.0)
device xhci # XHCI PCI->USB interface (USB 3.0)
device usb # USB Bus (required)
device ukbd # Keyboard
device umass # Disks/Mass storage - Requires scbus
and da
device virtio # Generic VirtIO bus (required)
device virtio_pci # VirtIO PCI device
device vtnet # VirtIO Ethernet device
device virtio_blk # VirtIO Block device
device virtio_scsi # VirtIO SCSI device
device virtio_balloon # VirtIO Memory Balloon device
device netmap # netmap(4) support
device crypto # Required by IPSEC
---------------------------------------------------------------
dmesg of 11.2 & upgraded to 11.3 CPU detection which matches my Opteron 6370p:
FreeBSD 11.2-RELEASE-p15 #0 r356556: Thu Jan 9 19:50:57 PST 2020
root at fbsd11-ovmf:/usr/obj/usr/src11.2/sys/Custom amd64
FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM
6.0.0)
VT(efifb): resolution 800x600
CPU: AMD Opteron 63xx class CPU (2003.51-MHz K8-class CPU)
Origin="AuthenticAMD" Id=0x600f20 Family=0x15 Model=0x2 Stepping=0
Features=0x1783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,HTT>
Features2=0xbeb83203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,SSE4.1,SSE4.2,x2APIC,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,HV>
AMD Features=0x26500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,LM>
AMD
Features2=0x210bf7<LAHF,CMP,SVM,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,XOP,FMA4,TBM>
Structured Extended Features=0x8<BMI1>
AMD Extended Feature Extensions ID EBX=0x1000
SVM: NAsids=16
TSC: P-state invariant
Hypervisor: Origin = "KVMKVMKVM"
FreeBSD 11.3-RELEASE-p5 #0 r356613: Sat Jan 11 15:56:34 PST 2020
root at fbsd11-ovmf:/usr/obj/usr/src11.3/sys/Custom amd64
FreeBSD clang version 8.0.0 (tags/RELEASE_800/final 356365) (based on LLVM
8.0.0)
VT(efifb): resolution 800x600
CPU: AMD Opteron 63xx class CPU (2004.09-MHz K8-class CPU)
Origin="AuthenticAMD" Id=0x600f20 Family=0x15 Model=0x2 Stepping=0
Features=0x1783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,HTT>
Features2=0xbeb83203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,SSE4.1,SSE4.2,x2APIC,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,HV>
AMD Features=0x26500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,LM>
AMD
Features2=0x210bf7<LAHF,CMP,SVM,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,XOP,FMA4,TBM>
Structured Extended Features=0x8<BMI1>
AMD Extended Feature Extensions ID EBX=0x1000
SVM: NAsids=16
TSC: P-state invariant
Hypervisor: Origin = "KVMKVMKVM"
cat /proc/cpuinfo (on host Ubuntu 18.0.4.3 LTS)
processor : 0
vendor_id : AuthenticAMD
cpu family : 21
model : 2
model name : AMD Opteron(tm) Processor 6370P
stepping : 0
<snip>
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb
rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid amd_dcm
aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave
avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse
3dnowprefetch osvw ibs xop skinit wdt fma4 tce nodeid_msr tbm topoext
perfctr_core perfctr_nb cpb hw_pstate ssbd ibpb vmmcall bmi1 arat npt lbrv
svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter
pfthreshold
bugs : fxsave_leak sysret_ss_attrs null_seg spectre_v1 spectre_v2
spec_store_bypass
<snip>
---------------------------------------------------------------
This VM was initially created to ensure that my proposed VirtIO patches would
also work in Q35 + OMVF for FreeBSD 12 (hence it's name: z_fbsd12_ovmf). I've
since used this configuration to install FreeBSD 11.2 to test viability of
backporting my proposed patches. The order of the devices seen in the command
line startup matches the order of customizations I've made/added in the GUI
virt-manager during the creation of the VM. Did you have a chane to test my
proposed VirtIO patches for FreeBSD 12.x within your environment? The VirtIO
devices should work along with SATA storage devices and em NICs. I'm unsure
about the VirtIO + netmap.
Regards,
Tommy
--
You are receiving this mail because:
You are the assignee for the bug.
More information about the freebsd-virtualization
mailing list