SMP, ACPI and interrupt storm

[Tech Lab Manager]

On Feb 4, 2008, at 6:00 AM, John Baldwin wrote:

> On Thursday 31 January 2008 02:35:52 pm Tech Lab Manager wrote:
>> On Jan 31, 2008, at 10:48 AM, Nate Lawson wrote:
>>
>>> Tech Lab Manager wrote:
>>>> Sorry for the cross-post from freebsd-smb.
>>>> Building 6.3-RELEASE and 7.0-RC1 on dual Xeon (4 CPU) boxes:
>>>>     options         SMP
>>>>     device          apic
>>>> SMP kernel builds fine, all 4 CPUs launch on reboot.
>>>> But I get a TON of interrupts from acpi0 -- about 67,000 per second
>>>> according to vmstat -i. With system at idle and almost no services
>>>> running, here is output of top -S:
>>>> last pid:   877;  load averages:  1.18,  0.48,  0.19
>>>> 75 processes:  6 running, 54 sleeping, 15 waiting
>>>> CPU states:  0.0% user,  0.0% nice,  0.2% system, 22.4%
>>>> interrupt,  77.4% idle
>>>> Mem: 31M Active, 12M Inact, 28M Wired, 16K Cache, 15M Buf, 3822M  
>>>> Free
>>>> Swap: 4096M Total, 4096M Free
>>>> PID USERNAME   THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU
>>>> COMMAND
>>>> 10 root         1 171   52     0K     8K RUN    3   1:11 99.18%
>>>> idle: cpu3
>>>> 13 root         1 171   52     0K     8K CPU0   0   1:10 98.88%
>>>> idle: cpu0
>>>> 12 root         1 171   52     0K     8K CPU1   1   1:09 98.78%
>>>> idle: cpu1
>>>> 21 root         1 -52 -171     0K     8K CPU2   2   0:54 87.24%
>>>> irq9: acpi0
>>>> 11 root         1 171   52     0K     8K RUN    2   0:17 11.19%
>>>> idle: cpu2
>>>> Notice high load and interrupt % of CPU.
>>>> If turn off ACPI (e.g. set hint.apic.0.disabled=1 in /boot/
>>>> loader.conf),
>>>> the interrupt storm ceases, but then I'm only running on one CPU.
>>>
>>> That doesn't turn off acpi, that turns of the APIC (interrupt
>>> controller).  Try:
>>>  hint.acpi.0.disabled=1
>>
>> Sorry, my mistake in writing ACPI above -- I *was* trying to turn off
>> apic, based on a note in the FreeBSD handbook.
>>
>> Disabling ACPI as you suggest above has the same effect as turning
>> off APIC: the interrupt storm is disabled but only one CPU is  
>> launched.
>>
>>>
>>>> The BIOS ACPI settings are all Enabled. Hyperthreading is Enabled.
>>>> These machines have been running RedHat Enterprise 5.0 with full
>>>> multiprocessor support.
>>>
>>> This looks like a failure to sleep in C1 (hlt).  Someone else
>>> reported this probably earlier, but all debugging showed the
>>> inexplicable -- the HLT instruction was being executed but just did
>>> not work (returned immediately).
>>>
>>> There will be a new 7.0 build that fixes one interrupt storm
>>> related to level-triggered GPEs.  If you can cvsup your 7.0 branch
>>> (RELENG_7_0) and retry, that might be helpful to see if it also
>>> fixes your problem.
>>
>> okay, I'm on RC1, will switch to RELENG and report back.
>>
>> I'm not sure if this is a red herring, but acpidump -t reports:
>>
>> Type=INT Override
>>          BUS=0
>>          IRQ=0
>>          INTR=2
>>          Flags={Polarity=conforming, Trigger=conforming}
>>
>> which looks wrong on several counts (IRQ, INTR should be 9,
>> Trigger=level). dmesg even says:
>> "MADT: Forcing active-low polarity and level trigger for SCI"
>
> No, this is an entry for something other than the SCI.  You can  
> have multiple
> interrupt override entries and this entry is typical on all x86  
> systems with
> APICs (the 8259As are tied into pin 0 as a daisy chain and IRQ0 is  
> tied into
> intpin 2 since IRQ2 isn't usable with 8259As.  Do you have an entry  
> at all
> for IRQ 9?  If not, then the hw.acpi.sci tunables currently won't  
> do anything
> (I can fix it so that they do, however).
>

Gotcha. (I'm really not much of a hardware guy so bear with me here.  
note: I csup'ed the RELENG_7_0 branch last Friday but no luck on this  
issue yet.)

Anyway, I see no IRQ 9 entry in acpidump, which I guess would explain  
why the attempt to set SCI tunables has so apparent effect.

# acpidump -t |less

/*
   RSD PTR: OEM=ACPIAM, ACPI_Rev=1.0x (0)
         RSDT=0xf7ff0000, cksum=64
  */
/*
   RSDT: Length=48, Revision=1, Checksum=38,
         OEMID=A M I, OEM Table ID=OEMRSDT, OEM Revision=0x4000308,
         Creator ID=MSFT, Creator Revision=0x97
         Entries={ 0xf7ff0200, 0xf7ff0300, 0xf7fff040 }
  */
/*
   FACP: Length=129, Revision=2, Checksum=65,
         OEMID=A M I, OEM Table ID=OEMFACP, OEM Revision=0x4000308,
         Creator ID=MSFT, Creator Revision=0x97
         FACS=0xf7fff000, DSDT=0xf7ff03a0
         INT_MODEL=APIC
         Preferred_PM_Profile=Unspecified (0)
         SCI_INT=9
         SMI_CMD=0xb2, ACPI_ENABLE=0xe1, ACPI_DISABLE=0x1e,  
S4BIOS_REQ=0x0
         PSTATE_CNT=0xe2
         PM1a_EVT_BLK=0x400-0x403
         PM1a_CNT_BLK=0x404-0x405
         PM2_CNT_BLK=0x420-0x420
         PM_TMR_BLK=0x408-0x40b
         GPE0_BLK=0x428-0x42b
         GPE1_BLK=0x42c-0x42f, GPE1_BASE=16
         CST_CNT=0xe3
         P_LVL2_LAT=101 us, P_LVL3_LAT=1001 us
         FLUSH_SIZE=1024, FLUSH_STRIDE=16
         DUTY_OFFSET=1, DUTY_WIDTH=3
         DAY_ALRM=13, MON_ALRM=0, CENTURY=0
         IAPC_BOOT_ARCH={LEGACY_DEV,8042}
         Flags={WBINVD,PROC_C1,P_LVL2_UP,SLP_BUTTON,RTC_S4,HEADLESS}
  */
/*
   FACS: Length=64, HwSig=0x00000000, Firm_Wake_Vec=0x00000000
         Global_Lock=
         Flags=
         Version=1
  */
/*
   DSDT: Length=12275, Revision=1, Checksum=151,
         OEMID=0ABBP, OEM Table ID=0ABBP000, OEM Revision=0x0,
         Creator ID=INTL, Creator Revision=0x2002026
  */
/*
   APIC: Length=146, Revision=1, Checksum=145,
         OEMID=A M I, OEM Table ID=OEMAPIC, OEM Revision=0x4000308,
         Creator ID=MSFT, Creator Revision=0x97
         Local APIC ADDR=0xfee00000
         Flags={PC-AT}

         Type=Local APIC
         ACPI CPU=1
         Flags={ENABLED}
         APIC ID=0

         Type=Local APIC
         ACPI CPU=2
         Flags={ENABLED}
         APIC ID=6

         Type=Local APIC
         ACPI CPU=3
         Flags={ENABLED}
         APIC ID=1

         Type=Local APIC
         ACPI CPU=4
         Flags={ENABLED}
         APIC ID=7

         Type=IO APIC
         APIC ID=8
         INT BASE=0
         ADDR=0x00000000fec00000

         Type=IO APIC
         APIC ID=9
         INT BASE=24
         ADDR=0x00000000fec80000

         Type=IO APIC
         APIC ID=10
         INT BASE=48
         ADDR=0x00000000fec80400

         Type=IO APIC
         APIC ID=11
         INT BASE=72
         ADDR=0x00000000fec81000

         Type=IO APIC
         APIC ID=12
         INT BASE=96
         ADDR=0x00000000fec81400

         Type=INT Override
         BUS=0
         IRQ=0
         INTR=2
         Flags={Polarity=conforming, Trigger=conforming}
  */
/*
   OEMB: Length=63, Revision=1, Checksum=73,
         OEMID=A M I, OEM Table ID=OEMBIOS, OEM Revision=0x4000308,
         Creator ID=MSFT, Creator Revision=0x97
  */