svn commit: r187880 - in head/sys: amd64/amd64 amd64/include i386/i386 i386/include

John Baldwin jhb at
Wed Jun 17 15:47:34 UTC 2009

On Thursday 29 January 2009 4:22:56 am Jeff Roberson wrote:
> Author: jeff
> Date: Thu Jan 29 09:22:56 2009
> New Revision: 187880
> URL:
> Log:
>    - Allocate apic vectors on a per-cpu basis.  This allows us to allocate
>      more irqs as we have more cpus.  This is principally useful on systems
>      with msi devices which may want many irqs per-cpu.
>   Discussed with:	jhb
>   Sponsored by:	Nokia

It turns out that this completely breaks devices using multiple MSI (not 
MSI-X) messages.  In MSI-X each interrupt has a separate address and data 
register, so each one can have an independent IDT vector and CPU.  This is 
not the case with MSI.  With MSI you only have a single address and data 
register set.  If you have multiple messages, the OS is required to leave the 
bottom N (log_2 (count)) bits set to zero and the individual message is 
encoded by via those bottom N bits.  This requires that multiple MSI messages 
be treated as a single group that all go to the same CPU.  Furthermore, the 
block of IDT vectors for the group has to be aligned on an N bit boundary.  
This was the purpose of the apic_alloc_vectors() function you removed.  I 
think that part of the change will need to be reverted.  msi_alloc() will 
probably need to preallocate vectors again rather than deferring them to 
msi_enable_intr().  Also, we can move MSI groups around.  I had changes in a 
p4 tree to basically fail to bind messages that weren't the first message in 
such a group.  Those can be revived for this purpose.

I think it is currently accidentally working for mav@'s AHCI tests because he 
does a bus_setup_intr() of all the interrupts in sequence during attach.  
That causes the IDT vectors to happen to all be contiguous on CPU 0 during 
boot.  It would fail if the driver was kldloaded.  Also, he must currently 
just be very lucky that the first IDT vector he allocates is properly 

John Baldwin

More information about the svn-src-all mailing list