Multiple MSI on SMP, misrouting or misunderstanding?

[Alexander Motin]

Hi.

While experimenting with using multiple MSIs support on AHCI controller 
I have got the problem. When system boots as UP - everything is fine, 
driver allocates all available 16 MSIs and works. But when system booted 
as SMP, interrupts begin to behave strange: I didn't receive expected 
AHCI IRQs, but instead receive IRQ1 interrupts of atkbd0, while I have 
no PS/2 keyboard/mouse attached.

As I have found, problem appears due to IRQ rebalancing between CPUs. As 
I have got, MSI requires that all vectors from the same group to be 
allocated sequentially, but IRQ rebalancing breaks correct order, that 
happed during initial allocation.

I was quite surprised by this issue. If multiple MSI vectors of the same 
device have to be allocated sequentially and bound to the same CPU, then 
they will be unable to give any SMP scalability benefits. Am I right, or 
there is some special technique expected to be used to somehow 
distribute grouped MSI vectors between CPUs which we don't have?

I have made small patch that denies rebalancing for grouped MSIs, to 
make them work at least somehow. It works fine for me, but I am not sure 
that it is the best solution.

-- 
Alexander Motin
-------------- next part --------------
--- msi.c.prev	2009-06-08 11:30:13.000000000 +0300
+++ msi.c	2009-06-08 11:30:06.000000000 +0300
@@ -210,6 +210,8 @@ msi_assign_cpu(struct intsrc *isrc, u_in
 	old_id = msi->msi_cpu;
 	if (old_vector && old_id == apic_id)
 		return;
+	if (old_vector && !msi->msi_msix && msi->msi_first->msi_count > 1)
+		return;
 	/* Allocate IDT vector on this cpu. */
 	vector = apic_alloc_vector(apic_id, msi->msi_irq);
 	if (vector == 0)