Xen (and others Hypervisors) how do they handle IPIs?

[Julian Elischer]

If CPUs are mapped around, how are IPIs handled? I assume they must be 
emulated?

I've noticed that under Xen (on both Amazon EC2 and a Redhat server) 
whenever you schedule a thread it always sits on the run queue for 20 
uSecs before it starts running. It looks to me like it's the IPI 
taking a long time to be emulated.

We have some workloads where there is a lot of flipping back and forth 
between threads and they are slowed down by an order of magnitude due 
to this..

turning off NOADAPTIVE(mumble) seems to help a bit as some of the 
reschedules go away, but it's still a problem.

Does anyone know if BHyVe or HyperV also have this problem? I have not 
yet gone to the source of Xen to see what it does but it would be 
educational to know what the other Hypervisors do.