157k interrupts per second causing 60% CPU load on idle system

[Matt Thyer]

On 21 March 2012 00:03, Gary Palmer <gpalmer at freebsd.org> wrote:

> On Tue, Mar 20, 2012 at 11:10:10PM +1030, Matt Thyer wrote:
> > On 20 March 2012 22:24, Ivan Voras <ivoras at freebsd.org> wrote:
> >
> > > On 20 March 2012 12:52, Matt Thyer <matt.thyer at gmail.com> wrote:
> > > > On 20 March 2012 21:12, Ivan Voras <ivoras at freebsd.org> wrote:
> > > >>
> > > >> On 20/03/2012 06:26, Matt Thyer wrote:
> > > >> > I've upgraded my FreeBSD-STABLE NAS from r225723 (22nd Sept 2011)
> to
> > > >> > r232477 (4th Mar 2012) and am finding that a system process called
> > > >> > "intr"
> > > >> > is now constantly using about 60% of 1 CPU starting a short time
> after
> > > >> > reboot (possibly triggered by use of the samba server).
> > > >> >
> > > >> > When this starts, systat -vm 1 says that the system is 85% idle
> and
> > > 14%
> > > >> > interrupt handling.
> > > >> > It says that there's around 157k interrupts per second.
> > > >> >
> > > >>
> > > >> Ok, but *which* interrupt is getting triggered? Please send the
> output
> > > >> of "vmstat -i".
> > > >>
> > > >>
> > > > interrupt                          total       rate
> > > > irq16: uhci0+                 3392184862     126692
> > >
> > > Ok, something's probably wrong with USB. Can you disable it in BIOS?
> > >
> > >
> > > > cpu0: timer                     53549677       1999
> > > > irq256: mps0                     2643187         98
> > > > irq257: re0                      5508108        205
> > > > irq258: ahci0                     160717          6
> > > > cpu1: timer                     53525300       1999
> > > > cpu2: timer                     53525300       1999
> > > > cpu3: timer                     53525296       1999
> > > > Total                         3614622447     134999
> > > >
> > >
> >
> > I did just update the BIOS at about the same time so the difference may
> be
> > due to that change.
> >
> > I'll try a few things such as:
> >
> > - Unplugging any USB things (I've only got a keyboard plugged in).
> > - Downgrade BIOS.
> >
> > I'll get back to you all soon.
>
> It would be interesting to know if there are other devices on irq16 also.
>
> grep 'irq 16' /var/run/dmesg.boot
>
> I think the '+' on the irq16 line from vmstat means the interrupt is
> shared, but the man page doesn't mention it so I'm not 100% sure
>
> Thanks,
>
> Gary
>

Good point...

pcib1: <PCI-PCI bridge> irq 16 at device 1.0 on pci0
mps0: <LSI SAS2008> port 0xee00-0xeeff mem
0xfbdfc000-0xfbdfffff,0xfbd80000-0xfbdbffff irq 16 at device 0.0 on pci1
vgapci0: <VGA-compatible display> port 0xff00-0xff07 mem
0xfb400000-0xfb7fffff,0xe0000000-0xefffffff irq 16 at device 2.0 on pci0
uhci0: <UHCI (generic) USB controller> port 0xfe00-0xfe1f irq 16 at device
26.0 on pci0
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0
atapci0: <JMicron JMB368 UDMA133 controller> port
0xdf00-0xdf07,0xde00-0xde03,0xdd00-0xdd07,0xdc00-0xdc03,0xdb00-0xdb0f irq
16 at device 0.0 on pci3

I'd suspect the SAS/SATA HBA using the mps0 driver as that's where I have
the raidz2 on 8 drives.

Is this still the old driver or has the new LSI authored driver been added
to -STABLE yet ?