IBM xSeries 336 dual Xeon hangs on boot when APIC enabled

Sun Aug 13 15:41:08 UTC 2006

I'm trying to boot FreeBSD 6.1-RELEASE/amd64 on an IBM xSeries 336 machine
with dual Xeons 3.2GHz installed.

The installation was successful, but
if I try to boot the SMP kernel, it hangs after detection of SCSI and ATA
devices (possibly when doing the initialization of the mpt0 RAID controller,
or when it tries to start the second CPU?).

boot -v doesn't give any error messages at that point. Disabling or enabling
hyperthreading does not make a difference. Disabling the APIC allows it to
boot, but gives me only one CPU. Apart from the missing CPU, everything
seems to work fine.
Doing a verbose boot with APIC disabled does generate a lot of these messages
(practically continuously):

SCSI IO Request @ 0xffffffffb3975880
Chain Offset 0x10
MsgFlags 0x00
MsgContext 0x000100f0
Bus: 0
TargetID 0
SenseBufferLength 32
LUN: 0x0
Control 0x01000000 WRITE SIMPLEQ
DataLength 0x00004000
SenseBufAddr 0xcf4d91e0
CDB[0:10] 2a 00 00 b2 73 9f 00 00 20 00
SE32 0xffffffffb39fe030: Addr=0x4ee31000 FlagsLength=0x14001000
HOST_TO_IOC
SE32 0xffffffffb39fe038: Addr=0x4eff2000 FlagsLength=0x94001000
HOST_TO_IOC LAST_ELEMENT
CE32 0xffffffffb39fe040: Addr=0xcf4d9048 NxtChnO=0x0 Flgs=0x30 Len=0x10
SE32 0xffffffffb39fe048: Addr=0x4ec93000 FlagsLength=0x14001000
HOST_TO_IOC
SE32 0xffffffffb39fe050: Addr=0x4edf4000 FlagsLength=0xd5001000
HOST_TO_IOC LAST_ELEMENT END_OF_BUFFER END_OF_LIST

Can this be related to the problem in any way? Attached is a dmesg from a
successful boot with APIC disabled, with a note of where the boot hangs when
APIC is enabled.

Does anyone have experience with these machines, or hints on what to try
next?

Arjan

dmesg.boot:
Copyright (c) 1992-2006 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD 6.1-RELEASE #0: Sun May 7 04:15:57 UTC 2006
root at bloom.cse.buffalo.edu:/usr/obj/usr/src/sys/SMP
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(TM) CPU 3.20GHz (3200.14-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0xf43 Stepping = 3
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x641d<SSE3,RSVD2,MON,DS_CPL,CNTX-ID,CX16,<b14>>
AMD Features=0x20000800<SYSCALL,LM>
real memory = 4966055936 (4736 MB)
avail memory = 4127162368 (3935 MB)
kbd1 at kbdmux0
acpi0: <IBM SERONYXP> on motherboard
acpi_bus_number: can't get _ADR
acpi_bus_number: can't get _ADR
acpi0: Power Button (fixed)
acpi_bus_number: can't get _ADR
acpi_bus_number: can't get _ADR
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x588-0x58b on acpi0
cpu0: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> on acpi0
pci0: <ACPI PCI bus> on pcib0
pci0: <unknown> at device 0.1 (no driver attached)
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci2: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci3: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 0.0 on pci3
pci4: <ACPI PCI bus> on pcib3
mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x4000-0x40ff mem
0xdeff0000-0xdeffffff,0xdefe0000-0xdefeffff irq 11 at device 1.0 on pci4
mpt0: [GIANT-LOCKED]
mpt0: MPI Version=1.2.15.0
mpt0: Unhandled Event Notify Frame. Event 0xa.
mpt0: Capabilities: ( RAID-1E RAID-1 SAFTE )
mpt0: 1 Active Volume (1 Max)
mpt0: 2 Hidden Drive Members (6 Max)
pcib4: <ACPI PCI-PCI bridge> at device 0.2 on pci3
pci5: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci6: <ACPI PCI bus> on pcib5
bge0: <Broadcom BCM5721 Gigabit Ethernet, ASIC rev. 0x4101> mem
0xdcff0000-0xdcffffff irq 11 at device 0.0 on pci6
miibus0: <MII bus> on bge0
brgphy0: <BCM5750 10/100/1000baseTX PHY> on miibus0
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX,
1000baseTX-FDX, auto
bge0: Ethernet address: 00:14:5e:7e:a4:0c
pcib6: <ACPI PCI-PCI bridge> at device 7.0 on pci0
pci7: <ACPI PCI bus> on pcib6
bge1: <Broadcom BCM5721 Gigabit Ethernet, ASIC rev. 0x4101> mem
0xdaff0000-0xdaffffff irq 11 at device 0.0 on pci7
miibus1: <MII bus> on bge1
brgphy1: <BCM5750 10/100/1000baseTX PHY> on miibus1
brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX,
1000baseTX-FDX, auto
bge1: Ethernet address: 00:14:5e:7e:a4:0d
pci0: <base peripheral> at device 8.0 (no driver attached)
uhci0: <Intel 82801EB (ICH5) USB controller USB-A> port 0x2200-0x221f irq 11
at device 29.0 on pci0
uhci0: [GIANT-LOCKED]
usb0: <Intel 82801EB (ICH5) USB controller USB-A> on uhci0
usb0: USB revision 1.0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1: <Intel 82801EB (ICH5) USB controller USB-B> port 0x2600-0x261f irq 3
at device 29.1 on pci0
uhci1: [GIANT-LOCKED]
usb1: <Intel 82801EB (ICH5) USB controller USB-B> on uhci1
usb1: USB revision 1.0
uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
ehci0: <Intel 82801EB/R (ICH5) USB 2.0 controller> mem 0xf9000000-0xf90003ff
irq 5 at device 29.7 on pci0
ehci0: [GIANT-LOCKED]
usb2: EHCI version 1.0
usb2: companion controllers, 2 ports each: usb0 usb1
usb2: <Intel 82801EB/R (ICH5) USB 2.0 controller> on ehci0
usb2: USB revision 2.0
uhub2: Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub2: 4 ports with 4 removable, self powered
pcib7: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci1: <ACPI PCI bus> on pcib7
pci1: <display, VGA> at device 1.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH5 SATA150 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x480-0x48f at device 31.2 on pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
atkbdc0: <Keyboard controller (i8042)> port 0x64,0x60 irq 1 on acpi0
atkbd0: <AT Keyboard> flags 0x1 irq 1 on atkbdc0
atkbd: unable to set the command byte.
device_attach: atkbd0 attach returned 6
psm0: unable to set the command byte.
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on
acpi0
sio0: type 16550A
orm0: <ISA Option ROM> at iomem 0xc0000-0xcafff on isa0
ppc0: cannot reserve I/O port range
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
uhub3: Mitsumi Electric Hub in Apple Extended USB Keyboard, class 9/0, rev
1.10/4.10, addr 2
uhub3: 3 ports with 2 removable, bus powered
ukbd0: Mitsumi Electric Apple Extended USB Keyboard, rev 1.10/4.10, addr 3,
iclass 3/1
kbd0 at ukbd0
uhid0: Mitsumi Electric Apple Extended USB Keyboard, rev 1.10/4.10, addr 3,
iclass 3/1
Timecounter "TSC" frequency 3200142440 Hz quality 800
Timecounters tick every 1.000 msec
module_register_init: MOD_LOAD (amr_linux, 0xffffffff8062a040, 0) error 6
Waiting 5 seconds for SCSI devices to settle
acd0: CDRW <UJDA765aDVD/CDRW/1.20> at ata0-master UDMA33
--- POINT WHERE BOOT HANGS IF APIC ENABLED ---
mpt0:vol0(mpt0:0:0): Settings ( Hot-Plug-Spares )
mpt0:vol0(mpt0:0:0): Using Spare Pool: 0
mpt0:vol0(mpt0:0:0): 2 Members:
(mpt0:0:0): Primary
(mpt0:0:1): Secondary
mpt0:vol0(mpt0:0:0): RAID-1 - Optimal
mpt0:vol0(mpt0:0:0): Status ( Enabled )
(mpt0:vol0:0): Physical (mpt0:0:0), Pass-thru (mpt0:1:0)
(mpt0:vol0:0): Online
(mpt0:vol0:1): Physical (mpt0:0:1), Pass-thru (mpt0:1:1)
(mpt0:vol0:1): Online
ses0 at mpt0 bus 0 target 8 lun 0
ses0: <IBM 25P3495a S320 1 1> Fixed Processor SCSI-2 device
ses0: 3.300MB/s transfers
ses0: SAF-TE Compliant Device
pass2 at mpt0 bus 1 target 0 lun 0
pass2: <IBM-ESXS VPR073C3-ETS10FN S410> Fixed unknown SCSI-4 device
pass2: 320.000MB/s transfers (160.000MHz, offset 127, 16bit), Tagged
Queueing Enabled
pass3 at mpt0 bus 1 target 1 lun 0
pass3: <IBM-ESXS VPR073C3-ETS10FN S410> Fixed unknown SCSI-4 device
pass3: 320.000MB/s transfers (160.000MHz, offset 127, 16bit), Tagged
Queueing Enabled
da0 at mpt0 bus 0 target 0 lun 0
da0: <LSILOGIC 1030 IM IM 1000> Fixed Direct Access SCSI-2 device
da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit), Tagged Queueing
Enabled
da0: 69878MB (143110144 512 byte sectors: 255H 63S/T 8908C)
Trying to mount root from ufs:/dev/da0s1a