12.0-RELEASE und kvm/qemu using <cpu mode='host-model' check='partial'> on AMD EPYC

Christian Kratzer ck-lists at cksoft.de
Mon Feb 11 13:32:11 UTC 2019


Hi,

I am running freebsd vm on debian 10 buster with libvirt/kvm/qemu.

I have several kvm hosts in the cluster.  Some with various intel xeon and others with AMD EPYC 7301 cpu.

FreeBSD vms upto 11.2-RELEASE-p9 boo fine on all systems when passing through the host cpu using following libvirt xml

   <cpu mode='host-model' check='partial'>
     <model fallback='allow'/>
   </cpu>

FreeBSD 12.0-RELEASE and later hang after boot when swithcing to usermode in start_init: trying /sbin/init

Following is dmesg from a succesfull boot of 12.0-RELEASE using host-model on Intel CPU

 	Copyright (c) 1992-2018 The FreeBSD Project.
 	Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
 		The Regents of the University of California. All rights reserved.
 	FreeBSD is a registered trademark of The FreeBSD Foundation.
 	FreeBSD 12.0-RELEASE-p3 GENERIC amd64
 	FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1)
 	VT(vga): text 80x25
 	CPU: QEMU Virtual CPU version 2.1.0 (2400.13-MHz K8-class CPU)
 	  Origin="GenuineIntel"  Id=0x663  Family=0x6  Model=0x6  Stepping=3
 	  Features=0x783fbfd<FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2>
 	  Features2=0x80a02001<SSE3,CX16,x2APIC,POPCNT,HV>
 	  AMD Features=0x20100800<SYSCALL,NX,LM>
 	  AMD Features2=0x1<LAHF>
 	Hypervisor: Origin = "KVMKVMKVM"
 	real memory  = 1073741824 (1024 MB)
 	avail memory = 1000058880 (953 MB)
 	Event timer "LAPIC" quality 100
 	ACPI APIC Table: <BOCHS  BXPCAPIC>
 	FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
 	FreeBSD/SMP: 4 package(s) x 1 core(s)
 	...

Following is dmesg from a succesfull boot of 12.0-RELEASE using host-model on the qemu virtual cpu


 	Copyright (c) 1992-2018 The FreeBSD Project.
 	Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
 		The Regents of the University of California. All rights reserved.
 	FreeBSD is a registered trademark of The FreeBSD Foundation.
 	FreeBSD 12.0-RELEASE-p3 GENERIC amd64
 	FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1)
 	VT(vga): text 80x25
 	CPU: QEMU Virtual CPU version 2.1.0 (2200.06-MHz K8-class CPU)
 	  Origin="AuthenticAMD"  Id=0x663  Family=0x6  Model=0x6  Stepping=3
 	  Features=0x783fbfd<FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2>
 	  Features2=0x80a02001<SSE3,CX16,x2APIC,POPCNT,HV>
 	  AMD Features=0x20100800<SYSCALL,NX,LM>
 	  AMD Features2=0x65<LAHF,SVM,ABM,SSE4A>
 	  SVM: NAsids=16
 	Hypervisor: Origin = "KVMKVMKVM"
 	real memory  = 4294967296 (4096 MB)
 	avail memory = 4099080192 (3909 MB)
 	Event timer "LAPIC" quality 100
 	ACPI APIC Table: <BOCHS  BXPCAPIC>
 	FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
 	FreeBSD/SMP: 4 package(s) x 1 core(s)

Following is dmesg from a succesfull boot of 11.2-RELEASE using host-model on AMD EPYC

 	Copyright (c) 1992-2018 The FreeBSD Project.
 	Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
 		The Regents of the University of California. All rights reserved.
 	FreeBSD is a registered trademark of The FreeBSD Foundation.
 	FreeBSD 11.2-RELEASE-p9 #0: Tue Feb  5 15:30:36 UTC 2019
 	    root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
 	FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0)
 	VT(vga): text 80x25
 	CPU: AMD EPYC Processor (with IBPB) (2200.05-MHz K8-class CPU)
 	  Origin="AuthenticAMD"  Id=0x800f12  Family=0x17  Model=0x1  Stepping=2
 	  Features=0x783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2>
 	  Features2=0xfff83203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
 	  AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>
 	  AMD Features2=0x8003f7<LAHF,CMP,SVM,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,PCXC>
 	  Structured Extended Features=0x201c01ab<FSGSBASE,TSCADJ,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,SHA>
 	  XSAVE Features=0x7<XSAVEOPT,XSAVEC,XINUSE>
 	  AMD Extended Feature Extensions ID EBX=0x2001000
 	  SVM: NAsids=16
 	Hypervisor: Origin = "KVMKVMKVM"
 	real memory  = 4294967296 (4096 MB)
 	avail memory = 4088360960 (3898 MB)
 	Event timer "LAPIC" quality 600
 	ACPI APIC Table: <BOCHS  BXPCAPIC>
 	FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
 	FreeBSD/SMP: 4 package(s)
 	...


And finally following is the failing verbose boot of 12.0-RELEASE on AMD-EPYC

 	Table 'FACP' at 0xbffe104e
 	Table 'APIC' at 0xbffe10c2
 	APIC: Found table at 0xbffe10c2
 	APIC: Using the MADT enumerator.
 	Copyright (c) 1992-2018 The FreeBSD Project.
 	Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
 		The Regents of the University of California. All rights reserved.
 	FreeBSD is a registered trademark of The FreeBSD Foundation.
 	FreeBSD 12.0-RELEASE-p3 GENERIC amd64
 	FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1)
 	Table 'FACP' at 0xbffe104e
 	Table 'APIC' at 0xbffe10c2
 	Table 'HPET' at 0xbffe1152
 	ACPI: No SRAT table found
 	PPIM 0: PA=0xb8000, VA=0xffffffff82810000, size=0x8000, mode=0
 	pmap: large map 8 PML4 slots (4096 Gb)
 	VT(vga): text 80x25
 	Preloaded elf kernel "/boot/kernel/kernel" at 0xffffffff8263e000.
 	Preloaded boot_entropy_cache "/boot/entropy" at 0xffffffff826471d0.
 	Table 'FACP' at 0xbffe104e
 	FACP: Found table at 0xbffe104e
 	Calibrating TSC clock ... TSC clock: 2200048246 Hz
 	CPU: AMD EPYC Processor (with IBPB) (2200.05-MHz K8-class CPU)
 	  Origin="AuthenticAMD"  Id=0x800f12  Family=0x17  Model=0x1  Stepping=2
 	  Features=0x783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2>
 	  Features2=0xfff83203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
 	  AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>
 	  AMD Features2=0x8003f7<LAHF,CMP,SVM,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,PCXC>
 	  Structured Extended Features=0x209c01ab<FSGSBASE,TSCADJ,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA>
 	  XSAVE Features=0x7<XSAVEOPT,XSAVEC,XINUSE>
 	  AMD Extended Feature Extensions ID EBX=0x2001000
 	  SVM: Features=0x0
 	Revision=1, ASIDs=16
 	L1 2MB data TLB: 255 entries, 1-way associative
 	L1 2MB instruction TLB: 255 entries, 1-way associative
 	L1 4KB data TLB: 255 entries, 1-way associative
 	L1 4KB instruction TLB: 255 entries, 1-way associative
 	L1 data cache: 64 kbytes, 64 bytes/line, 1 lines/tag, 2-way associative
 	L1 instruction cache: 64 kbytes, 64 bytes/line, 1 lines/tag, 2-way associative
 	L2 2MB unified TLB: 0 entries, disabled/not present
 	L2 4KB data TLB: 512 entries, 4-way associative
 	L2 4KB instruction TLB: 512 entries, 4-way associative
 	L2 unified cache: 512 kbytes, 64 bytes/line, 1 lines/tag, 16-way associative
 	Hypervisor: Origin = "KVMKVMKVM"
 	real memory  = 4294967296 (4096 MB)
 	Physical memory chunk(s):
 	0x0000000000001000 - 0x000000000009efff, 647168 bytes (158 pages)
 	0x0000000000103000 - 0x00000000001fffff, 1036288 bytes (253 pages)
 	0x0000000002800000 - 0x00000000b7d44fff, 3042201600 bytes (742725 pages)
 	0x0000000100000000 - 0x000000013ffe7fff, 1073643520 bytes (262120 pages)
 	avail memory = 4099080192 (3909 MB)
 	Table 'FACP' at 0xbffe104e
 	Table 'APIC' at 0xbffe10c2
 	Table 'HPET' at 0xbffe1152
 	ACPI: No DMAR table found
 	...
 	...
 	...
 	ata0: stat0=0x00 err=0x00 lsb=0x00 msb=0x00
 	ata0: stat1=0x00 err=0x00 lsb=0x00 msb=0x00
 	ata0: reset tp2 stat0=00 stat1=00 devices=0x0
 	ata1: reset tp1 mask=03 ostat0=00 ostat1=00
 	ata1: stat0=0x00 err=0x00 lsb=0x00 msb=0x00
 	ata1: stat1=0x00 err=0x00 lsb=0x00 msb=0x00
 	ata1: reset tp2 stat0=00 stat1=00 devices=0x0
 	Trying to mount root from ufs:/dev/vtbd0p2 [rw]...
 	atrtc0: providing initial system time
 	start_init: trying /sbin/init
 	uhub0: 2 ports with 2 removable, self powered

After that the system freezes.

I can provide more complete information if somebody tells me what would be helpfull.

How should I go about debugging this ???

Greetings
Christian

-- 
Christian Kratzer                   CK Software GmbH
Email:   ck at cksoft.de               Wildberger Weg 24/2
Phone:   +49 7032 893 997 - 0       D-71126 Gaeufelden
Fax:     +49 7032 893 997 - 9       HRB 245288, Amtsgericht Stuttgart
Mobile:  +49 171 1947 843           Geschaeftsfuehrer: Christian Kratzer
Web:     http://www.cksoft.de/


More information about the freebsd-virtualization mailing list