kern/113138: interrupt storm on 6.x kernels on an MS-1029 (AMD64/ATI)

Tue May 29 23:00:12 UTC 2007

>Number:         113138
>Category:       kern
>Synopsis:       interrupt storm on 6.x kernels on an MS-1029 (AMD64/ATI)
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue May 29 23:00:11 GMT 2007
>Closed-Date:
>Last-Modified:
>Originator:     Robert Johnson
>Release:        6.1
>Organization:
>Environment:
FreeBSD  6.1-RELEASE FreeBSD 6.1-RELEASE #0: Sun May  7 04:04:14 UTC 2006     root at bloom.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64
>Description:

STATEMENT OF PROBLEM

====================

FreeBSD 6.1 runs very slowly out of the box on my MS-1029 AMD Turion notebook computer.  The first line from the output of 'ps -aux' tells that interrupt 19, in use by the ohci and cardbus systems, is consuming 80% of the CPU time:

USER    PID %CPU %MEM   VSZ   RSS  TT  STAT STARTED      TIME COMMAND

root     21 79.1  0.0     0    16  ??  LL   11:52AM  25:48.51 [irq19: cbb0 ohci

ATTEMPTED RESOLUTIONS

=====================

I attempted to disable the affected devices in the boot configuration file "/boot/device.hints."  In all cases my 'hints' were ignored and the device drivers were loaded as normal.  I also tried to do the equivalent with sysctls, also to no avail (though I might be doing the Wrong Thing):

hint.ohci.0.disabled="1"

hint.usb.0.disabled="1"

hint.cardbus.0.disabled="1"

hint.pccard.0.disabled="1"

Booting in safe mode and with ACPI disabled did not help.

I rebuilt the kernel with the cardbus driver disabled.  This caused the cardbus to disappear from the problem interrupt but the ohci driver was still using irq19 and was having an interrupt storm.  In the compromised state of my computer it takes 48 hours to compile a kernel, I don't have access to another computer and anyways I need USB support, so I don't want to recompile again without USB support.

I installed FreeBSD 6.0 and also 6.2; the problem was the same.  I installed FreeBSD 5.3 and there was no problem (except that I needed to disable ACPI), nor was there a problem with NetBSD 3.0, nor Linux 2.4 or 2.6.  OpenBSD 3.9 also appears to suffer from an interrupt problem.

DETAILED INFORMATION

====================

The computer is a Microstar MS-1029 with an AMD MT-37 Turion processor running at 2.0ghz.

The Motherboard uses an ATI Radeon Xpress 200P Chipset (North Bridge RX480M, South Bridge SB400).

The Graphics Card is an ATI Mobility Radeon X700 with 128 MB of dedicated RAM.

The system has 1024 MB of memory.  The BIOS is "A1029AMS v2.70"

I used the minimal installation from 6.1-RELEASE-amd64-disc1.iso

# uname -a

FreeBSD  6.1-RELEASE FreeBSD 6.1-RELEASE #0: Sun May  7 04:04:14 UTC 2006     root at bloom.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64

The output from "dmesg -a" s at the bottom of this report.

Windows XP reports the following information regarding IRQ 19:

IRQ 19	Standard OpenHCD USB Host Controller	OK

IRQ 19	Standard OpenHCD USB Host Controller	OK

IRQ 19	Standard Enhanced PCI to USB Host Controller	OK

IRQ 19	Ricoh R/RL/5C476(II) or Compatible CardBus Controller	OK

The results of running other operating systems on the computer:

FreeBSD 6.1 -!	ohci (USB) and cardbus interrupt storm on IRQ 19
FreeBSD 6.2 -!  identical problem

FreeBSD 6.0 -!	identical problem

FreeBSD 5.3 --	ACPI crashes on boot.  No interrupt storm when booted

		without ACPI, kernel ignores spurious interrupt on IRQ 7

NetBSD 3.0 --	no problem

OpenBSD 3.9 -!	Apparently the same problem as with the FreeBSD 6

		kernels... interrupts use 80+% of the cpu; the individual

		interrupts are not listed by ps... I don't know much about

		OpenBSD so I can't say more.

Slackware 9.1 with Linux 2.4 kernel -- no interrupt problem (but disk

		access is slow, probably an old ATA driver)

Slackware 11 with Linux 2.6 kernel -- no problem

WindowsXP Professional with SP2 -- occasional spurious 'data'

		returned by ACPI hardware, runs fine otherwise.

When I first installed FreeBSD 6.1, the install program ran fine, and at the end of the install process I checked the state of the processes using the holographic shell; there was no interrupt storm.

When I tried booting my system using the install disk as a rescue disk, the system suffered from the same interrupt storm (perhaps there is a significant difference between the "install configuration" and the "rescue configuration")?

I noticed in the boot process that the computer appears to run fine through the initial boot, but at around the point that it begins processing the rc files it slows down.  The slowdown appears to begin between 
"module_register_init: MOD_LOAD (amr_linux, 0xffffffff806205d0, 0) error 6"
 and "Entropy Harvesting:";
In particular, the slowdown first appears noticeable when the ATA devices were probed (but this may just be due to waiting for devices to respond).  Following up on this observation, I disabled entropy harvesting of interrupts but this did not help.  Windows reports this about the IDE/ATA system:

Name	Standard Dual Channel PCI IDE Controller

Manufacturer	(Standard IDE ATA/ATAPI controllers)

Status	OK

PNP Device ID	PCI\VEN_1002&DEV_4376&SUBSYS_02911462&REV_00\3&267A616A&0&A1

I/O Port	0x0000FF00-0x0000FF0F

Driver	c:\windows\system32\drivers\pciide.sys (5.1.2600.0 (XPClient.010817-1148), 3.25 KB (3,328 bytes), 8/3/2004 9:07 PM)

Windows also reports an occasional error from the ACPI hardware:

\Device\ACPIEC: The embedded controller (EC) hardware didn't respond within the timeout period.  This may indicate an error in the EC hardware or firmware, or possibly a poorly designed BIOS which accesses the EC in an unsafe manner.  The EC driver will retry the failed transaction if possible.

This is all the headway I've been able to make, sorry I wasn't able to pinpoint the problem.

# dmesg -a
Copyright (c) 1992-2006 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD 6.1-RELEASE #0: Sun May  7 04:04:14 UTC 2006
    root at bloom.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Turion(tm) 64 Mobile Technology MT-37 (1989.82-MHz K8-class CPU)
  Origin = "AuthenticAMD"  Id = 0x20f42  Stepping = 2
  Features=0x78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2>
  Features2=0x1<SSE3>
  AMD Features=0xe2500800<SYSCALL,NX,MMX+,FFXSR,LM,3DNow+,3DNow>
  AMD Features2=0x1<LAHF>
real memory  = 536084480 (511 MB)
avail memory = 505630720 (482 MB)
ACPI APIC Table: <MSI    OEMAPIC >
ioapic0: Changing APIC ID to 1
ioapic0 <Version 2.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <MSI 1029> on motherboard
acpi0: Overriding SCI Interrupt from IRQ 9 to IRQ 21
acpi0: Power Button (fixed)
acpi_ec0: <Embedded Controller: GPE 0x6> port 0x62,0x66 on acpi0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_perf0: invalid _PSS package
acpi_throttle0: <ACPI CPU Throttling> on cpu0
acpi_throttle0: CLK_VAL field overlaps THT_EN bit
device_attach: acpi_throttle0 attach returned 6
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pci1: <display, VGA> at device 0.0 (no driver attached)
ohci0: <ATI SB400 USB Controller> mem 0xfbdfd000-0xfbdfdfff irq 19 at device 19.0 on pci0
ohci0: [GIANT-LOCKED]
usb0: OHCI version 1.0, legacy support
usb0: <ATI SB400 USB Controller> on ohci0
usb0: USB revision 1.0
uhub0: ATI OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 4 ports with 4 removable, self powered
ohci1: <ATI SB400 USB Controller> mem 0xfbdfe000-0xfbdfefff irq 19 at device 19.1 on pci0
ohci1: [GIANT-LOCKED]
usb1: OHCI version 1.0, legacy support
usb1: <ATI SB400 USB Controller> on ohci1
usb1: USB revision 1.0
uhub1: ATI OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 4 ports with 4 removable, self powered
ehci0: <ATI SB400 USB 2.0 controller> mem 0xfbdff000-0xfbdfffff irq 19 at device 19.2 on pci0
ehci0: [GIANT-LOCKED]
usb2: EHCI version 1.0
usb2: companion controllers, 4 ports each: usb0 usb1
usb2: <ATI SB400 USB 2.0 controller> on ehci0
usb2: USB revision 2.0
uhub2: ATI EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub2: 8 ports with 8 removable, self powered
pci0: <serial bus, SMBus> at device 20.0 (no driver attached)
atapci0: <ATI IXP400 UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff00-0xff0f at device 20.1 on pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
isab0: <PCI-ISA bridge> at device 20.3 on pci0
isa0: <ISA bus> on isab0
pcib2: <ACPI PCI-PCI bridge> at device 20.4 on pci0
pci2: <ACPI PCI bus> on pcib2
rl0: <RealTek 8139 10/100BaseTX> port 0xe800-0xe8ff mem 0xfbfffc00-0xfbfffcff irq 18 at device 3.0 on pci2
miibus0: <MII bus> on rl0
rlphy0: <RealTek internal media interface> on miibus0
rlphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
rl0: Ethernet address: 00:13:d3:af:0a:b0
cbb0: <RF5C476 PCI-CardBus Bridge> irq 19 at device 4.0 on pci2
cardbus0: <CardBus bus> on cbb0
pccard0: <16-bit PCCard bus> on cbb0
cbb1: <RF5C476 PCI-CardBus Bridge> irq 20 at device 4.1 on pci2
cardbus1: <CardBus bus> on cbb1
pccard1: <16-bit PCCard bus> on cbb1
fwohci0: <Ricoh R5C552> mem 0xfbfff000-0xfbfff7ff irq 21 at device 4.2 on pci2
fwohci0: OHCI version 1.0 (ROM=1)
fwohci0: No. of Isochronous channels is 4.
fwohci0: EUI64 00:10:dc:00:00:d9:2b:8b
fwohci0: Phy 1394a available S400, 2 ports.
fwohci0: Link S400, max_rec 2048 bytes.
firewire0: <IEEE1394(FireWire) bus> on fwohci0
fwe0: <Ethernet over FireWire> on firewire0
if_fwe0: Fake Ethernet address: 02:10:dc:d9:2b:8b
fwe0: Ethernet address: 02:10:dc:d9:2b:8b
fwe0: if_start running deferred for Giant
sbp0: <SBP-2/SCSI over FireWire> on firewire0
fwohci0: Initiate bus reset
fwohci0: node_id=0xc800ffc0, gen=1, CYCLEMASTER mode
firewire0: 1 nodes, maxhop <= 0, cable IRM = 0 (me)
firewire0: bus manager 0 (me)
ral0: <Ralink Technology RT2500> mem 0xfbffc000-0xfbffdfff irq 16 at device 9.0 on pci2
ral0: MAC/BBP RT2560 (rev 0x04), RF RT2525
ral0: Ethernet address: 00:11:09:2a:ae:b0
pci0: <multimedia, audio> at device 20.5 (no driver attached)
pci0: <simple comms, generic modem> at device 20.6 (no driver attached)
acpi_lid0: <Control Method Lid Switch> on acpi0
acpi_button0: <Sleep Button> on acpi0
acpi_button1: <Power Button> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> flags 0x1 irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse, device ID 3
acpi_acad0: <AC Adapter> on acpi0
battery0: <ACPI Control Method Battery> on acpi0
orm0: <ISA Option ROM> at iomem 0xce800-0xcf7ff on isa0
ppc0: cannot reserve I/O port range
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 8250 or not responding
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounter "TSC" frequency 1989818882 Hz quality 800
Timecounters tick every 1.000 msec
module_register_init: MOD_LOAD (amr_linux, 0xffffffff806205d0, 0) error 6
ad0: 95396MB <HTS541010G9AT00 MBZOA60A> at ata0-master UDMA100
acd0: DVDR <HL-DT-ST DVD-RW GWA-4082N/CB03> at ata1-master UDMA33
Trying to mount root from ufs:/dev/ad0s2a
Loading configuration files.
Entropy harvesting:
 interrupts
 ethernet
 point_to_point
 kickstart
>How-To-Repeat:
>Fix:
>Release-Note:
>Audit-Trail:
>Unformatted: