SunFire x4275

[Jesse Kempf]

Hi,
We recently got in a SunFire x4275, one of Sun's new Nehalem boxes. The 
full server architecture whitepaper is here: 
https://www.sun.com/offers/details/X4x70_server_architecture.html 
(requires registration).

The upshot of the system is that there are a pair of "intelligent" 
risers in the machine, which take a x8 PCIe link and mux it to a pair of 
x8 PCIe links. Sun uses the IDT PES24T6G2 PCIe switch to do the muxing.

FreeBSD, when it boots, can't see past the risers. This holds for both 
7.2p1 and 8-CURRENT. It can see the IDT PCIe switches, and if a PCIe 
card is moved off the intelligent riser, to the one dumb riser in the 
machine, the card can be seen. Interestingly, when I boot FreeBSD with 
ACPI turned off, all the devices past the risers can be seen, but device 
attachment fails spectacularly.

I've tried setting debug.acpi.disabled for individual subsystems that 
make sense, and have not been able to come up with a combination that 
allows the system to boot as well as see past the risers. From what I 
understand, this is not too surprising.

When I say "can't see past the risers", I mean:

pcib9: <ACPI PCI-PCI bridge> at device 0.0 on pci25
pcib9:   domain            0
pcib9:   secondary bus     26
pcib9:   subordinate bus   38
pcib9:   I/O decode        0x8000-0x9fff
pcib9:   memory decode     0xfaa00000-0xfabfffff
pcib9:   no prefetched decode
device_attach: pcib9 attach returned 6

is what the kernel spits out when I do a verbose boot. None of the PCI 
buses past the riser are enumerated. I dumped the ACPI AML block and 
decompiled it. The only errors that iasl spit out when I recompiled it 
was use of the reserved variables '_T_0' and '_T_1'. Switching them to 
'LOL0' and 'LOL1', recompiling, and telling the loader where to find the 
new blob did not help.

I am, at this point, at a loss for what to do next.

Here's the dmesg from the verbose boot:
http://www.cs.rpi.edu/~kempfj2/x4275.dmesg (GENERIC-DMESG simply ups the 
kernel message buffer to 100 pages from 10 so that I could capture 
everything into /var/run/boot.dmesg).

The ASL:
http://www.cs.rpi.edu/~kempfj2/x4275.asl

And pciconf -vl:
http://www.cs.rpi.edu/~kempfj2/x4275pci.txt

sysctl hw.acpi:
http://www.cs.rpi.edu/~kempfj2/x4275hw.acpi

Boot -v with ACPI disabled explodes on discovering igb0 with a page 
fault in kernel mode:
http://www.cs.rpi.edu/~kempfj2/x4275.verbose-noacpi

And this is a non-verbose boot with acpi disabled:
http://www.cs.rpi.edu/~kempfj2/x4275.noacpi


What's the next step in debugging this, and how can I be of assistance?

Thanks,
-Jesse Kempf



------------------------------------------------------------------------
The information contained in this communication is intended
only for the use of the recipient(s) named above. It may
contain information that is privileged or confidential, and
may be protected by State and/or Federal Regulations. If
the reader of this message is not the intended recipient,
you are hereby notified that any dissemination,
distribution, or copying of this communication, or any of
its contents, is strictly prohibited. If you have received
this communication in error, please return it to the sender
immediately and delete the original message and any copy
of it from your computer system. If you have any questions
concerning this message, please contact the sender.
------------------------------------------------------------------------