kernel: interrupt storm detected on "irq22:"; throttling interrupt source [7.0-RELEASE]

[John Baldwin]

On Monday 22 September 2008 11:09:52 am Oliver Peter wrote:
> On Tue, Sep 09, 2008 at 02:24:16PM -0400, John Baldwin wrote:
> > On Monday 08 September 2008 07:08:05 pm Oliver Peter wrote:
> > > ...
> > > ... of course I could do that - but could you please be so kind and
> > > explain how to do that?  :-)
> > 
> > Well, you could start by adding a printf to ata's interrupt handler, but 
that 
> > would result in a flood on your screen.  You could maybe use a KTR 
instead, 
> > but only do it if the interrupt handler doesn't find any work to do (e.g. 
no 
> > pending request) perhaps.  I think the ATA interrrupt handler already 
reads a 
> > status register of some sort, but I could be wrong.
> 
> I feels like I need a rosetta stone for that...  :)
> 
> This is my "production" machine in the datacenter 2,000km away -
> I would love to debug that interrupt storm with KTR, but I'm not sure
> what I'm doing at all and nobody can ensure that my machine will
> come up again after that...
> 
> Anyway I would like to learn more about the KTR stuff.
> I've never heard about that before, is that a good point to start out?
> 
>      http://www.watson.org/~robert/freebsd/netperf/ktr/

There's also a manpage (man ktr).  You probably want to just enable KTR_DRIVER 
in KTR_COMPILE and KTR_MASK and add custom traces to the ata driver.  Adding 
traces is about like printf:

	CTR0(KTR_DRIVER, "a message");
	CTR1(KTR_DRIVER, "with one argument %d", foo);
	CTR2(KTR_DRIVER, "two args: %p (%d)", req, req->count);

etc.

-- 
John Baldwin