Xen (and others Hypervisors) how do they handle IPIs?

[Roger Pau Monné]

On 27/09/13 09:44, Julian Elischer wrote:
> On 9/26/13 5:50 PM, Roger Pau Monné wrote:
>> On 26/09/13 03:48, Julian Elischer wrote:
>>> On 9/25/13 11:04 PM, Roger Pau Monné wrote:
>>>> On 25/09/13 16:56, Julian Elischer wrote:
>>>>> If CPUs are mapped around, how are IPIs handled? I assume they must be
>>>>> emulated?
>>>>>
>>>>> I've noticed that under Xen (on both Amazon EC2 and a Redhat server)
>>>>> whenever you schedule a thread it always sits on the run queue for 20
>>>>> uSecs before it starts running. It looks to me like it's the IPI
>>>>> taking
>>>>> a long time to be emulated.
>>>> This has been improved on the FreeBSD Xen PVHVM port by using PV IPIs
>>>> instead of the emulated ones, see r255331. It should be faster than the
>>>> previous emulated implementation.
>>>>
>>> I missed that.. thanks!
>>> Do you (or anyone else) know if this can be used on Amazon EC2?
>>> And do you need a specific version/configuration of Xen to be able to
>>> use it?
> 
> I ran a new GENERIC kernel on an Amazon Xen 4.2 system and it performed
> a bit better.
> delays in the running of threads reduced from a constant 20uSecs to a
> variable amount between 1.5uSecs and 9uSecs.
> It's still very slow when compared with real hardware.
> (timing viewed using Schedgraph)
> 
> Is there a way to see whether PV IPIs are active?
> (it looks it from the results but that could be so many other things too).

The easiest way to see if PV IPIs are active is to run vmstat -i inside
of the guest and search for the following lines:

irq836: cpu6:r                         2          0
irq839: cpu6:irg                    2530         70
irq841: cpu6:b                      2175         60

This is the number of the different kinds of PV IPIs received on each
CPU. You can find the correlation between this names and the IPIs on top
of the file x86/xen/hvm.c.

Roger.