Multiple MSI on SMP, misrouting or misunderstanding?
mav at FreeBSD.org
Mon Jun 8 09:15:39 UTC 2009
While experimenting with using multiple MSIs support on AHCI controller
I have got the problem. When system boots as UP - everything is fine,
driver allocates all available 16 MSIs and works. But when system booted
as SMP, interrupts begin to behave strange: I didn't receive expected
AHCI IRQs, but instead receive IRQ1 interrupts of atkbd0, while I have
no PS/2 keyboard/mouse attached.
As I have found, problem appears due to IRQ rebalancing between CPUs. As
I have got, MSI requires that all vectors from the same group to be
allocated sequentially, but IRQ rebalancing breaks correct order, that
happed during initial allocation.
I was quite surprised by this issue. If multiple MSI vectors of the same
device have to be allocated sequentially and bound to the same CPU, then
they will be unable to give any SMP scalability benefits. Am I right, or
there is some special technique expected to be used to somehow
distribute grouped MSI vectors between CPUs which we don't have?
I have made small patch that denies rebalancing for grouped MSIs, to
make them work at least somehow. It works fine for me, but I am not sure
that it is the best solution.
-------------- next part --------------
--- msi.c.prev 2009-06-08 11:30:13.000000000 +0300
+++ msi.c 2009-06-08 11:30:06.000000000 +0300
@@ -210,6 +210,8 @@ msi_assign_cpu(struct intsrc *isrc, u_in
old_id = msi->msi_cpu;
if (old_vector && old_id == apic_id)
+ if (old_vector && !msi->msi_msix && msi->msi_first->msi_count > 1)
/* Allocate IDT vector on this cpu. */
vector = apic_alloc_vector(apic_id, msi->msi_irq);
if (vector == 0)
More information about the freebsd-current