Ryzen lockup on bhyve was (Re: new Ryzen lockup issue ?)

[Mike Tancsa]

On 2/25/2018 12:51 PM, Peter Grehan wrote:
> Hi Mike,
> 
>  (moving this to -virtualization)
> 
>> Actually I can confirm the same sort of hard lockup happens on my Epyc
>> board with RELENG11.  It also happens in current. I will file a PR and
>> post on freebsd-current in case someone has any suggestions on how to
>> try and figure out whats going on.
> 
>  I could reproduce this on a Ryzen 1700 with HT enabled - it took ~700
> seconds, and the entire machine locked up.
> 
>  An interesting point is a single VM was able to run to completion.
> 
>  I'll keep testing with various combinations of numbers of VMs,
> different vCPUs per VM, and pinning guest vCPUs.

Hi Peter,
	I upgraded both boxes (Ryzen and Epyc) to HEAD.  For the record, the
Epyc CPU is

CPU1 Information
AMD EPYC 7281 16-Core Family: 17h Model: 00h-0Fh
Microcode Patch Level: 8001207
Bios Release Date: 01/08/2018
SuperMicro: H11SSL-i

Not sure if its related to core count or not, but it takes several
instances on the Epyc to lock up and takes a lot longer. I enabled IOMMU
on it as well, and it's dmesg shows below.  But the lockups are still
there with or without it enabled.

ivhd0: <AMD-Vi/IOMMU ivhd with EFR> on acpi0
ivhd0: Flag:b0<IotlbSup,Coherent>
ivhd0: Features(type:0x11) MsiNumPPR = 0 PNBanks= 2 PNCounters= 0
ivhd0: Extended features[31:0]:22294ada<PPRSup,NXSup,GTSup,IASup> HATS =
0x2 GATS = 0x0 GLXSup = 0x1 SmiFSup = 0x1 SmiFRC = 0x2 GAMSup = 0x1
DualPortLogSup = 0x2 DualEventLogSup = 0x2
ivhd0: Extended features[62:32]:f77ef<USSup> Max PASID: 0x2f
DevTblSegSup = 0x3 MarcSup = 0x1
ivhd0: supported paging level:7, will use only: 4
ivhd0: device range: 0x0 - 0xffff
ivhd0: PCI cap 0x190b640f at 0x40 feature:19<IOTLB,EFR,CapExt>
ivhd1: <AMD-Vi/IOMMU ivhd with EFR> on acpi0
ivhd1: Flag:b0<IotlbSup,Coherent>
ivhd1: Features(type:0x11) MsiNumPPR = 0 PNBanks= 2 PNCounters= 0
ivhd1: Extended features[31:0]:22294ada<PPRSup,NXSup,GTSup,IASup> HATS =
0x2 GATS = 0x0 GLXSup = 0x1 SmiFSup = 0x1 SmiFRC = 0x2 GAMSup = 0x1
DualPortLogSup = 0x2 DualEventLogSup = 0x2
ivhd1: Extended features[62:32]:f77ef<USSup> Max PASID: 0x2f
DevTblSegSup = 0x3 MarcSup = 0x1
ivhd1: supported paging level:7, will use only: 4
ivhd1: device range: 0x0 - 0xffff
ivhd1: PCI cap 0x190b640f at 0x40 feature:19<IOTLB,EFR,CapExt>
ivhd2: <AMD-Vi/IOMMU ivhd with EFR> on acpi0
ivhd2: Flag:b0<IotlbSup,Coherent>
ivhd2: Features(type:0x11) MsiNumPPR = 0 PNBanks= 2 PNCounters= 0
ivhd2: Extended features[31:0]:22294ada<PPRSup,NXSup,GTSup,IASup> HATS =
0x2 GATS = 0x0 GLXSup = 0x1 SmiFSup = 0x1 SmiFRC = 0x2 GAMSup = 0x1
DualPortLogSup = 0x2 DualEventLogSup = 0x2
ivhd2: Extended features[62:32]:f77ef<USSup> Max PASID: 0x2f
DevTblSegSup = 0x3 MarcSup = 0x1
ivhd2: supported paging level:7, will use only: 4
ivhd2: device range: 0x0 - 0x3ffe
ivhd2: PCI cap 0x190b640f at 0x40 feature:19<IOTLB,EFR,CapExt>
ivhd3: <AMD-Vi/IOMMU ivhd with EFR> on acpi0
ivhd3: Flag:b0<IotlbSup,Coherent>
ivhd3: Features(type:0x11) MsiNumPPR = 0 PNBanks= 2 PNCounters= 0
ivhd3: Extended features[31:0]:22294ada<PPRSup,NXSup,GTSup,IASup> HATS =
0x2 GATS = 0x0 GLXSup = 0x1 SmiFSup = 0x1 SmiFRC = 0x2 GAMSup = 0x1
DualPortLogSup = 0x2 DualEventLogSup = 0x2
ivhd3: Extended features[62:32]:f77ef<USSup> Max PASID: 0x2f
DevTblSegSup = 0x3 MarcSup = 0x1
ivhd3: supported paging level:7, will use only: 4
ivhd3: device range: 0x0 - 0x3ffe
ivhd3: PCI cap 0x190b640f at 0x40 feature:19<IOTLB,EFR,CapExt>

	---Mike



-- 
-------------------
Mike Tancsa, tel +1 519 651 3400 x203
Sentex Communications, mike at sentex.net
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada