Bheve: Slow linux syscalls on AMD

[Willem Jan Withagen]

On 10-6-2014 20:04, Peter Grehan wrote:
> Hi Willem,
> 
>> I've got KTR compiled in, but last time I switched it on.
>> I got swamped in traffic, and I sort of got locked out of the
>> server... :(
>> Could also be because I was writing it to a file as well.
>>
>> So you'll have to help/tell me what to do.
> 
>  For looking at this, I'd use one single-vCPU guest to avoid KTR noise
> from other vCPU/guests. The problem shows up with this so no issue there.

The trace consists mainly of large number of sequeces like: (ktrdump -trc)
----
   204   1            37052 vm ubuntu-14.04[0]: pending intr 65
   203   1            39489 vm ubuntu-14.04[0]: VMEXIT halted CPU.
   202   1            53836 vm ubuntu-14.04[0]: SVM:Guest in interrupt
shadow.

   201   1            69072 vm ubuntu-14.04[0]: SVM:Enter vmrun
RIP:0xffffffff8104f596 inst len=0/15

   200   1            37218 vm ubuntu-14.04[0]: pending intr 65
   199   1            39534 vm ubuntu-14.04[0]: VMEXIT halted CPU.
----

intermixed with blocks:
----
 13485   1            65347 vm ubuntu-14.04[0]: SVM:event
injected,vector=65.

 13484   1            41457 vm ubuntu-14.04[0]: vlapic_update_ppr 0x40
 13483   1            53987 vm ubuntu-14.04[0]: vlapic_intr_accepted
isr7 0x00000000
 13482   1            52734 vm ubuntu-14.04[0]: vlapic_intr_accepted
isr6 0x00000000
 13481   1            52729 vm ubuntu-14.04[0]: vlapic_intr_accepted
isr5 0x00000000
 13480   1            52784 vm ubuntu-14.04[0]: vlapic_intr_accepted
isr4 0x00000000
 13479   1            52755 vm ubuntu-14.04[0]: vlapic_intr_accepted
isr3 0x00000000
 13478   1            52840 vm ubuntu-14.04[0]: vlapic_intr_accepted
isr2 0x00000002
 13477   1            52709 vm ubuntu-14.04[0]: vlapic_intr_accepted
isr1 0x00000000
 13476   1            52669 vm ubuntu-14.04[0]: vlapic_intr_accepted
isr0 0x00000000
 13475   1            52722 vm ubuntu-14.04[0]: vlapic_intr_accepted
irr7 0x00000000
 13474   1            52694 vm ubuntu-14.04[0]: vlapic_intr_accepted
irr6 0x00000000
 13473   1            52701 vm ubuntu-14.04[0]: vlapic_intr_accepted
irr5 0x00000000
 13472   1            52705 vm ubuntu-14.04[0]: vlapic_intr_accepted
irr4 0x00000000
 13471   1            52692 vm ubuntu-14.04[0]: vlapic_intr_accepted
irr3 0x00000000
 13470   1            52884 vm ubuntu-14.04[0]: vlapic_intr_accepted
irr2 0x00000000
 13469   1            52839 vm ubuntu-14.04[0]: vlapic_intr_accepted
irr1 0x00000000
 13468   1            53055 vm ubuntu-14.04[0]: vlapic_intr_accepted
irr0 0x00000000
----

So it looks like the VM goes halted, but interrupt 65 tries to wake it.
Dmesg however does not give a irq 65 in the guest.
wjw at ubuntu-14:~$ dmesg | grep irq
[  0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
[  0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
[  0.000000] nr_irqs_gsi: 40
[  0.000000] NR_IRQS:16640 nr_irqs:256 16
[  0.321489] hpet: hpet2 irq 40 for MSI
[  0.748675] 00:01: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is
a 16550A
[  0.770813] 00:02: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is
a 16550A
[  0.778181] virtio-pci 0000:00:02.0: irq 41 for MSI/MSI-X
[  0.778203] virtio-pci 0000:00:02.0: irq 42 for MSI/MSI-X
[  0.778224] virtio-pci 0000:00:02.0: irq 43 for MSI/MSI-X
[  1.393123] ahci 0000:00:03.0: irq 44 for MSI/MSI-X
[  1.424971] ata1: SATA max UDMA/133 abar m1024 at 0xc0002000 port
0xc0002100 irq 44
[  1.435304] ahci 0000:00:04.0: irq 45 for MSI/MSI-X
[  1.456767] ata7: SATA max UDMA/133 abar m1024 at 0xc0002400 port
0xc0002500 irq 45

A larger ktrdump is at:
	http://smtp.digiware.nl/tmp/ktrdump.log

--WjW