Weird PCI interrupt delivery problem

Thu Dec 1 18:42:30 GMT 2005

On Wednesday 30 November 2005 12:23 pm, Craig Boston wrote:
> On Wed, Nov 30, 2005 at 10:35:13AM -0500, John Baldwin wrote:
> > > With both ACPI & APIC enabled, it only lasts a few seconds.
> >
> > You didn't have to futz with the routing in this case?
>
> No, I just took the default routing specified by ACPI.

Ok.

> > > With ACPI disabled, the system panics because the mptable is broken.
> > > However, I was able to hack the kernel to override the mptable and
> > > route the interrupts to the correct pins.
> >
> > You know that you can override individual routings just using tunables
> > without having to hack the table.  Just use something like:
>
> Oh yeah, thanks, I forgot about that.  Though I think I would have to
> hack the table anyway just to get the system booted -- it panics due to
> the bogus entries long before it tries to route any INTs.

Ah.

> A GENERIC kernel with ACPI but no APIC does the same thing, however.
> Everything ends up on IRQ 11 and it does just that much quicker.
> vmstat -i shows IRQ 11 frozen at 55 interrupts (the number varies per
> boot).

Ok.

> > > The final thing I tried is both APIC & ACPI disabled
> >
> > Do you have a dmesg from this?  Preferably a verbose one to see if
> > your $PIR has routing info for cbb0.
>
> Yes, I have verbose dmesgs up at
> http://www.gank.org/freebsd/l25/
>
> (I've been running RELENG_6, but reverted to a 6.0 GENERIC to create
> these as to rule out any local patches)
>
> 8259.txt is the one with both APIC & ACPI disabled.  I don't see a $PIR
> table anywhere though -- I don't think this board was intended to be
> run without ACPI...

Yeah, odd, no $PIR.  You can get your cbb0 to work though using a tunable in 
the loader.  Something like:

hw.pci9.1.INTA.irq=11

should do the trick to get your box working in the !ACPI and !APIC case at 
least and you can see if it has the same issue with freezing up then.

> > We mask the IRQs in the PIC while their ithread runs.  If your routing
> > is all screwed up that might result in the problems you are seeing.
> > Can you boot into Windows and jot down the IRQs it uses for each
> > device
>
> I'm pretty sure the routing is OK, but am game to try anything that
> might help.  I don't have Windows on this machine anymore, so I called a
> friend with an identical laptop and had him go through device manager.
> Highlights include USB on IRQ 19, the integrated RealTek NIC on IRQ 21,
> and the MiniPCI wireless on IRQ 22.  It matched exactly with with the
> IRQ lines that BSD uses with APIC, with or without ACPI.

Hmm, ok.

> > and then (if you are up to it), provide verbose dmesg's of an
> > unpatched kernel for the 4 cases of + ACPI + APIC, - ACPI + APIC, +
> > ACPI - APIC, - ACPI - APIC?
>
> Sure, see the link above.  The files are:
> 8259.txt	- ACPI - APIC
> acpi+apic.txt	+ ACPI + APIC
> acpi.txt	+ ACPI - APIC
>
> I couldn't get one for - ACPI + APIC as it panics on bootup and this
> machine doesn't have a serial port.  I did post a verbose boot with my
> patched kernel for that under apic+patches.txt.
>
> Also in there is mptable.txt, which shows the broken mptable.
> mptable-fixed.txt is the output of the mptable command when running with
> a patched kernel in APIC (no ACPI) mode.

Heh, yes, that's a screwed up MP table alright.  Hmm.  I think I may have 
figured out your problem.  FreeBSD's PCI bus is allocating the same resources 
to cbb0 that ath0 is already setup to use.  This is probably Warner's 
fault. :-/ (cc'd)  Your !ACPI and !APIC case is probably working because cbb0 
didn't attach.  For a test, you can try compiling cbb out of your kernel and 
seeing if the box works ok.

Note in your ACPI + APIC dmesg:

pcib2: <ACPI PCI-PCI bridge> at device 20.4 on pci0
pcib2:   secondary bus     9
pcib2:   subordinate bus   14
pcib2:   I/O decode        0xa000-0xafff
pcib2:   memory decode     0xd0200000-0xd02fffff
pcib2:   prefetched decode 0xfff00000-0xfffff
pcib2:   Subtractively decoded bridge.

Since this is subtractice, does this even need mem decode ranges setup?

pci9: <ACPI PCI bus> on pcib2
pci9: physical bus=9
found->	vendor=0x104c, dev=0xac56, revid=0x00
	bus=9, slot=1, func=0
	class=06-07-00, hdrtype=0x02, mfdev=0
	cmdreg=0x0000, statreg=0x0210, cachelnsz=0 (dwords)
	lattimer=0x20 (960 ns), mingnt=0x40 (16000 ns), maxlat=0x03 (750 ns)
	intpin=a, irq=255
	powerspec 2  supports D0 D1 D2 D3  current D0
	map[10]: type 1, range 32, base 00000000, size 12, memory disabled

That's cbb0 and it has no address assigned to BAR 0x10 (nor an IRQ).

Resources for re0:

found->	vendor=0x10ec, dev=0x8139, revid=0x10
	bus=9, slot=2, func=0
	map[10]: type 4, range 32, base 0000a000, size  8, enabled
pcib2: (null) requested I/O range 0xa000-0xa0ff: in range
	map[14]: type 1, range 32, base d0210000, size  8, enabled
pcib2: (null) requested memory range 0xd0210000-0xd02100ff: good

Resources for ath0:

found->	vendor=0x168c, dev=0x001a, revid=0x01
	bus=9, slot=4, func=0
	map[10]: type 1, range 32, base d0200000, size 16, enabled
pcib2: (null) requested memory range 0xd0200000-0xd020ffff: good

Then when cbb0 attaches we try to give it some resources:

cbb0: <TI1510 PCI-CardBus Bridge> at device 1.0 on pci9
pcib2: cbb0 requested memory range 0xd0200000-0xd02fffff: good
cbb0: Lazy allocation of 0x1000 bytes rid 0x10 type 3 at 0xd0211000

Hmm, maybe the message is just confusing and it gets resources at 0xd0211000 
which would be ok.  Hmm, back to square 1 I guess.  I'd still be interested 
to hear how the broken kernels fair if you take cbb out and how the 8259 
kernel does with the variable set to fix cbb0's interrupt.  You could also 
try moving one of the links to IRQ 10 in the ACPI case via a tunable such as:

hw.acpi.LNKA.irq=10 or some such (can't recall if that's the right name).

-- 
John Baldwin <jhb at FreeBSD.org>  <><  http://www.FreeBSD.org/~jhb/
"Power Users Use the Power to Serve"  =  http://www.FreeBSD.org