bhyve/amd: interrupt delivered when it shouldn't be?

[Andriy Gapon]

On 30/12/2017 01:45, Peter Grehan wrote:
> Hi Andriy,
> 
>> The hardware is AMD.
> 
>  Ryzen ?

No, a pretty old family 10h.

>> But what I see suggests that at this point a Local APIC timer interrupt gets
>> delivered to the thread.  And that causes all the mess as the thread holding the
>> spinlock gets preempted.
>>
>> Does this ring a bell to anyone?
> 
>  I have seen something similar to this after about ~20 mins when doing a current
> -j 16 buildworld in a guest, with the symptom being a spinlock timeout, with one
> vCPU spinning in
> 
> smp_targeted_tlb_shootdown() at smp_targeted_tlb_shootdown+0x352/frame
> 0xfffffe02c80098d0
> smp_masked_invlpg() at smp_masked_invlpg+0x4c/frame 0xfffffe02c8009900
> pmap_invalidate_page() at pmap_invalidate_page+0x191/frame 0xfffffe02c8009950
> pmap_ts_referenced() at pmap_ts_referenced+0x7b3/frame 0xfffffe02c8009a00
> vm_pageout() at vm_pageout+0xe04/frame 0xfffffe02c8009a70
> 
> ... and all the others eventually spinning on that held lock.

This looks very similar to what I see as well.

> However, NMIs are still able to get through (the post-panic ddb NMI IPI) so the
> VM isn't completely locked up - either an interrupt is missed, or a write isn't
> seen by the vCPU issuing the tlb shootdown.

Yes, I can also enter ddb and remote kgdb.

>> Is there any suspect code?
> 
>  Not sure yet, but the interrupt-injection path could do with a close inspection.
> 
>> It seems that we set v_intr_masking bit, so the rFLAGS / eFLAGS should be
>> completely virtualized.  So, maybe a hardware issue?
> 
>  Hard to say. Running with all vCPUs pinned makes the problem go away, but that
> could just mean the issue is isolated to when vCPUs migrate.

I was going to try pinned CPUs and wired memory next, so your observation
narrows the problem scope.

Thank you!


-- 
Andriy Gapon