PCI-Express interrupt issues

[Jason Harmening]

Hi All,

I'm the maintainer for the FreeBSD cx88 driver (multimedia/cx88).  I've
recently encountered some issues w/ interrupt handling, specifically on
PCI-Express systems, and I was hoping someone would be able to help me.

Issue #1:

The cx88 driver has split interrupt handling between filters and
ithreads since filters became available w/ 7.0.  For the past year, the
filters have been set up to return FILTER_SCHEDULE_THREAD if the status
register indicates an interrupt, FILTER_STRAY otherwise.  Everything
has worked fine. 

However, I recently stumbled across some documentation indicating that
FILTER_SCHEDULE_THREAD shouldn't be returned alone--instead, the filter
should return FILTER_HANDLED | FILTER_SCHEDULE_THREAD.  So I modified
the cx88 driver to do that instead, and that's where things turned
strange:

On machine #1, which uses a VIA K8T800 chipset, everything still worked
fine.

On machine #2, which uses a VIA K8T890 chipset, the threaded interrupt
handlers were no longer invoked.  It's as though the bus driver saw
FILTER_HANDLED in the bitmask and assumed the interrupt was already
processed without checking to see if an ithread should be scheduled.
What's interesting is that the only significant difference between the
K8T800 in machine #1 and the K8T890 in machine #2 is that the K8T890
supports PCI-Express, while the K8T800 is PCI-only.


Issue #2:

We're working on adding support for a newer generation of PCI-Express
cx88 devices, which support message signaled interrupts.
When MSIs are enabled for these boards, the interrupts seem to simply
stop firing after the first several.  During transfers across the
cx88's onboard I2C bus, the first several (e.g. 8-10) bytes will be
transferred successfully, which means the I2C transfer interrupts are
working for those bytes.  But after that the passive thread will timeout
waiting to be signaled by the interrupt handler.  The I2C interrupts
are handled entirely in their own filter (no ithread here), and they can
occur with high frequency (e.g. several hundred per second for large
firmware loads). We don't see anything in the system log indicating
that an interrupt storm is detected or that throttling is coming into
play here.

If we revert to legacy INTx interrupts, everything works fine.  All
other transfer logic is identical between the legacy and MSI cases.
The MSI behavior is identical between machine #2 w/ the K8T890 and a
third machine w/ an nForce4.  We haven't been able to try it anywhere
else yet.


I'm hoping someone will be able to shed some light on these
issues--neither one is a real emergency, but they're both kind of
troubling.

Thanks,
Jason