PCI-Express interrupt issues

Jason Harmening jason.harmening at gmail.com
Mon Feb 9 08:11:28 PST 2009


On Mon, Feb 9, 2009 at 8:46 AM, John Baldwin <jhb at freebsd.org> wrote:
> On Sunday 08 February 2009 12:32:02 pm Jason Harmening wrote:
>> Hi All,
>>
>> I'm the maintainer for the FreeBSD cx88 driver (multimedia/cx88).  I've
>> recently encountered some issues w/ interrupt handling, specifically on
>> PCI-Express systems, and I was hoping someone would be able to help me.
>>
>> Issue #1:
>>
>> The cx88 driver has split interrupt handling between filters and
>> ithreads since filters became available w/ 7.0.  For the past year, the
>> filters have been set up to return FILTER_SCHEDULE_THREAD if the status
>> register indicates an interrupt, FILTER_STRAY otherwise.  Everything
>> has worked fine.
>>
>> However, I recently stumbled across some documentation indicating that
>> FILTER_SCHEDULE_THREAD shouldn't be returned alone--instead, the filter
>> should return FILTER_HANDLED | FILTER_SCHEDULE_THREAD.  So I modified
>> the cx88 driver to do that instead, and that's where things turned
>> strange:
>>
>> On machine #1, which uses a VIA K8T800 chipset, everything still worked
>> fine.
>>
>> On machine #2, which uses a VIA K8T890 chipset, the threaded interrupt
>> handlers were no longer invoked.  It's as though the bus driver saw
>> FILTER_HANDLED in the bitmask and assumed the interrupt was already
>> processed without checking to see if an ithread should be scheduled.
>> What's interesting is that the only significant difference between the
>> K8T800 in machine #1 and the K8T890 in machine #2 is that the K8T890
>> supports PCI-Express, while the K8T800 is PCI-only.
>
> Are you seeing this only on 7.0?  Also, do you have 'INTR_FILTER' enabled in
> the kernel?  If you don't, then your ithread will never be called if you have
> a filter (actually, the bus_setup_intr() should fail in that case if you
> specify both).

All machines are running 7.1-STABLE.  Filters are enabled across the
board, and reverting to just returning  FILTER_SCHEDULE_THREAD fixes
the issue.

>
>> Issue #2:
>>
>> We're working on adding support for a newer generation of PCI-Express
>> cx88 devices, which support message signaled interrupts.
>> When MSIs are enabled for these boards, the interrupts seem to simply
>> stop firing after the first several.  During transfers across the
>> cx88's onboard I2C bus, the first several (e.g. 8-10) bytes will be
>> transferred successfully, which means the I2C transfer interrupts are
>> working for those bytes.  But after that the passive thread will timeout
>> waiting to be signaled by the interrupt handler.  The I2C interrupts
>> are handled entirely in their own filter (no ithread here), and they can
>> occur with high frequency (e.g. several hundred per second for large
>> firmware loads). We don't see anything in the system log indicating
>> that an interrupt storm is detected or that throttling is coming into
>> play here.
>>
>> If we revert to legacy INTx interrupts, everything works fine.  All
>> other transfer logic is identical between the legacy and MSI cases.
>> The MSI behavior is identical between machine #2 w/ the K8T890 and a
>> third machine w/ an nForce4.  We haven't been able to try it anywhere
>> else yet.
>>
>>
>> I'm hoping someone will be able to shed some light on these
>> issues--neither one is a real emergency, but they're both kind of
>> troubling.
>
> The only thing I can think of is perhaps a hardware bug.  MSI interrupts are
> edge triggered while INTx are level.  However, if a device implements MSI
> by "listening" to the level-triggered interrupt and sending a message when
> the level changes and it ever has a bug where it fails to send a message,
> then it won't recover.  FreeBSD doesn't do any masking of interrupts for MSI,
> it just installs the handler in the IDT and calls it for every interrupt.
> Thus, if you are not seeing interrupts it is likely not an issue with FreeBSD
> itself, but an issue in the hardware (or possibly the driver if your
> interrupt handler doesn't fully ACK something perhaps).

With this hardware, failing to ACK would just cause a storm, which
we'd see w/ INTx as well.   But you are right about the possibility of
a hardware issue--the particular piece we're testing right now has
some other known issues as well.  There's a linux driver that seems to
work OK w/ MSIs, but they don't actually use hardware I2C and don't
use interrupts there.  This issue just needs more testing w/ more
hardware, and we have a device hint to enable/disable MSIs anyway.

>
> --
> John Baldwin
>


More information about the freebsd-drivers mailing list