interrupt storm on ahci

[Steve Wills]

On Wed, Jun 11, 2014 at 09:18:17AM +0000, Steve Wills wrote:
> On Sat, Jun 07, 2014 at 02:53:05PM -0700, Neel Natu wrote:
> > Hi Steve,
> > 
> > On Sat, Jun 7, 2014 at 2:28 PM, Peter Grehan <grehan at freebsd.org> wrote:
> > > Hi Steve,
> > >
> > >
> > >> I'm running a FreeBSD guest in bhyve on a FreeBSD host. Both are running
> > >> FreeBSD CURRENT, r266947. I've gotten this message about 12 times since
> > >> boot:
> > >>
> > >> interrupt storm detected on "irq268:"; throttling interrupt source
> > >>
> > >> vmstat -i shows:
> > >>
> > >> irq268: ahci1                  236514222        839
> > >>
> > >> ahci1 is the second disk connected to the system:
> > >>
> > >> ahci1: <Intel ICH8 AHCI SATA controller> mem 0xc0002400-0xc00027ff irq 18
> > >> at device 4.0 on pci0
> > >>
> > >> The VM itself runs poudriere and was building a bunch of packages. At the
> > >> moment, the VM seems to be in a rather odd state. The poudriere jails are
> > >> running, but not doing anything. Ideas?
> > >
> > >
> > >  Is this an 8.* host ? I don't believe AHCI has MSI support on that version,
> > > and AHCI legacy interrupts haven't had a huge amount of testing under load.
> > >
> > >  If it is 8.*, I'd recommend using virtio-blk for the block device until we
> > > can work out what's going wrong.
> > >
> > 
> > The KTR trace would be useful to figure out what's happening.
> > 
> > To do that you can compile the host kernel and vmm.ko with the
> > following options:
> > options KTR
> > options KTR_MASK=(KTR_GEN)
> > options KTR_ENTRIES=(4*1024*1024)
> > 
> > And when you see the interrupt storm message in the guest you can execute:
> > sudo ktrdump -cto /tmp/ktrdump.out
> > 
> 
> This was added to the kernel config, the kernel was built and installed and the 
> host was rebooted. I started the VM and almost as soon as the workload was put
> back, the messages started again. So, I ran the ktrdump command above. I'll
> mail you privately with the location of the output.

Replying to myself just to note for the record that the interrupt storm was
harmless, because the throttling only reduced performance (I thought another
issue I was having at the same time was a result, but it was just a
coincidence) and that the default threshold for interrupt storms of 1000
interrupts per second may be too low in some cases, such as this one. I
disabled the throttling by setting the hw.intr_storm_threshold sysctl to 0 and
haven't had any issues.

Steve