svn commit: r187880 - in head/sys: amd64/amd64 amd64/include i386/i386 i386/include

[John Baldwin]

On Thursday 29 January 2009 4:22:56 am Jeff Roberson wrote:
> Author: jeff
> Date: Thu Jan 29 09:22:56 2009
> New Revision: 187880
> URL: http://svn.freebsd.org/changeset/base/187880
> 
> Log:
>    - Allocate apic vectors on a per-cpu basis.  This allows us to allocate
>      more irqs as we have more cpus.  This is principally useful on systems
>      with msi devices which may want many irqs per-cpu.
>   
>   Discussed with:	jhb
>   Sponsored by:	Nokia

It turns out that this completely breaks devices using multiple MSI (not 
MSI-X) messages.  In MSI-X each interrupt has a separate address and data 
register, so each one can have an independent IDT vector and CPU.  This is 
not the case with MSI.  With MSI you only have a single address and data 
register set.  If you have multiple messages, the OS is required to leave the 
bottom N (log_2 (count)) bits set to zero and the individual message is 
encoded by via those bottom N bits.  This requires that multiple MSI messages 
be treated as a single group that all go to the same CPU.  Furthermore, the 
block of IDT vectors for the group has to be aligned on an N bit boundary.  
This was the purpose of the apic_alloc_vectors() function you removed.  I 
think that part of the change will need to be reverted.  msi_alloc() will 
probably need to preallocate vectors again rather than deferring them to 
msi_enable_intr().  Also, we can move MSI groups around.  I had changes in a 
p4 tree to basically fail to bind messages that weren't the first message in 
such a group.  Those can be revived for this purpose.

I think it is currently accidentally working for mav@'s AHCI tests because he 
does a bus_setup_intr() of all the interrupts in sequence during attach.  
That causes the IDT vectors to happen to all be contiguous on CPU 0 during 
boot.  It would fail if the driver was kldloaded.  Also, he must currently 
just be very lucky that the first IDT vector he allocates is properly 
aligned.

-- 
John Baldwin