PCI-Express interrupt issues

John Baldwin jhb at freebsd.org
Mon Feb 9 07:53:36 PST 2009


On Sunday 08 February 2009 12:32:02 pm Jason Harmening wrote:
> Hi All,
> 
> I'm the maintainer for the FreeBSD cx88 driver (multimedia/cx88).  I've
> recently encountered some issues w/ interrupt handling, specifically on
> PCI-Express systems, and I was hoping someone would be able to help me.
> 
> Issue #1:
> 
> The cx88 driver has split interrupt handling between filters and
> ithreads since filters became available w/ 7.0.  For the past year, the
> filters have been set up to return FILTER_SCHEDULE_THREAD if the status
> register indicates an interrupt, FILTER_STRAY otherwise.  Everything
> has worked fine. 
> 
> However, I recently stumbled across some documentation indicating that
> FILTER_SCHEDULE_THREAD shouldn't be returned alone--instead, the filter
> should return FILTER_HANDLED | FILTER_SCHEDULE_THREAD.  So I modified
> the cx88 driver to do that instead, and that's where things turned
> strange:
> 
> On machine #1, which uses a VIA K8T800 chipset, everything still worked
> fine.
> 
> On machine #2, which uses a VIA K8T890 chipset, the threaded interrupt
> handlers were no longer invoked.  It's as though the bus driver saw
> FILTER_HANDLED in the bitmask and assumed the interrupt was already
> processed without checking to see if an ithread should be scheduled.
> What's interesting is that the only significant difference between the
> K8T800 in machine #1 and the K8T890 in machine #2 is that the K8T890
> supports PCI-Express, while the K8T800 is PCI-only.

Are you seeing this only on 7.0?  Also, do you have 'INTR_FILTER' enabled in 
the kernel?  If you don't, then your ithread will never be called if you have 
a filter (actually, the bus_setup_intr() should fail in that case if you 
specify both).

> Issue #2:
> 
> We're working on adding support for a newer generation of PCI-Express
> cx88 devices, which support message signaled interrupts.
> When MSIs are enabled for these boards, the interrupts seem to simply
> stop firing after the first several.  During transfers across the
> cx88's onboard I2C bus, the first several (e.g. 8-10) bytes will be
> transferred successfully, which means the I2C transfer interrupts are
> working for those bytes.  But after that the passive thread will timeout
> waiting to be signaled by the interrupt handler.  The I2C interrupts
> are handled entirely in their own filter (no ithread here), and they can
> occur with high frequency (e.g. several hundred per second for large
> firmware loads). We don't see anything in the system log indicating
> that an interrupt storm is detected or that throttling is coming into
> play here.
> 
> If we revert to legacy INTx interrupts, everything works fine.  All
> other transfer logic is identical between the legacy and MSI cases.
> The MSI behavior is identical between machine #2 w/ the K8T890 and a
> third machine w/ an nForce4.  We haven't been able to try it anywhere
> else yet.
> 
> 
> I'm hoping someone will be able to shed some light on these
> issues--neither one is a real emergency, but they're both kind of
> troubling.

The only thing I can think of is perhaps a hardware bug.  MSI interrupts are 
edge triggered while INTx are level.  However, if a device implements MSI 
by "listening" to the level-triggered interrupt and sending a message when 
the level changes and it ever has a bug where it fails to send a message, 
then it won't recover.  FreeBSD doesn't do any masking of interrupts for MSI, 
it just installs the handler in the IDT and calls it for every interrupt.  
Thus, if you are not seeing interrupts it is likely not an issue with FreeBSD 
itself, but an issue in the hardware (or possibly the driver if your 
interrupt handler doesn't fully ACK something perhaps).

-- 
John Baldwin


More information about the freebsd-drivers mailing list