IPMI During Boot

Tue Mar 14 19:06:41 UTC 2006

Hello,

I've tried to be fairly thorough in researching the problem before posting
to the list, however, there doesn't seem to be a great multitude of
information out there regarding IPMI configuration on FreeBSD.

Here's the setup:  I have a SuperMicro 6014H-82 Chassis with the
AOC-IPMI20-E
(http://supermicro.com/products/accessories/addon/AOC-IPMI20-E.cfm).  The
motherboard has an Intel adapter that uses the em(4) driver (so it isn't
suppose to suffer the same issues as the bge adapter).  I have ensured the
MAC address configured for the IPMI matches the MAC address of the network
card itself.  I'm using a private network IP (192.168.2.41) on the IPMI card
itself, and I've tried various settings for the adapter in the OS (I've
tried it using the same private IP, and a different private IP).  The IPMI
function itself works great, I can connect and perform the needed tasks, I
can even use Serial-On-Lan to get into the BIOS, the RAID card setup, as
well as log into the Operating system.  I've also configured the boot
settings so that I do get the menu of boot options, and I can see the boot
happening.

However, here's the issue:  During booting, when the "em" driver loads, the
IPMI function effectively ceases.  From the time the em driver is loaded,
until the OS actually configures the network adapter, I do not receive
output to the serial console.  If I disconnect the IPMI session, I cannot
reconnect during the time either (IPMI is suppose to connect at all times,
even when the box is powered off!).  If I don't disconnect, eventually, all
the backlog will spew to the screen and I'll be at the login prompt.
However, this poses a problem if I need to use single-user mode or if FSCK
requires some intervention.  Since single-user mode never configures the
network adapter, then it just never comes back.  Regarding FSCK, if a box
crashes and goes to FSCK, even if its set to automatically fsck (too screwed
up for background fsck or something), I have no way of seeing that, since it
occurs before the network adapter is configured as well.  This kinda defeats
the purpose of the IPMI card.

I'm using the SuperMicro IPMIView program.  I have tried both the regular
IPMI firmware that uses a non-standard RMCP protocol, as well as their
"beta" firmware that uses the standard RMCP+ protocol.  Using the RMCP+
firmware enables me to use ipmitool or such from command line, but the same
problem exists.

I have tried a multitude of things.  I've upgraded to the latest stable
version (6-STABLE).  Built the latest boot blocks (cd /usr/src/sys/boot &&
make && make install), installed them (fdisk -B && disklabel -B
/dev/aacd0s1).  Below are the various configs and messages I can think of to
post.  The kernel is pretty much GENERIC with maxusers, smp, and quota.

/boot.config:
-Dh -S19200

/boot/loader.conf:
boot_serial="YES"
boot_multicons="YES"
console="comconsole,vidconsole"
comconsole_speed="19200"

I'm pretty sure the above is overkill.  I've tried using just one or the
other or both, and while the console redirection still works, it still
freezes when the em driver loads.

One thing to note when look at the dmesg.  I do see the dmesg load fast on
the SOL up until the em loads, and then it shows this:

em0: Ethernet address: 00:30:48:2f:51:d2

Until the network interface is brought up in the later boot stages, at which
point it flies through what it missed to the login prompt.

/var/run/dmesg.boot:
Copyright (c) 1992-2006 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD 6.1-PRERELEASE #3: Tue Mar 14 13:35:52 EST 2006
    root at mysql02.secureserverdot.com:/usr/obj/usr/src/sys/SH
ACPI APIC Table: <PTLTD          APIC  >
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2800.12-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0xf43  Stepping = 3

Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA
,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x641d<SSE3,RSVD2,MON,DS_CPL,CNTX-ID,CX16,<b14>>
  AMD Features=0x20100800<SYSCALL,NX,LM>
  Hyperthreading: 2 logical CPUs
real memory  = 5100273664 (4864 MB)
avail memory = 4122357760 (3931 MB)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  6
 cpu3 (AP): APIC ID:  7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
ioapic2 <Version 2.0> irqs 48-71 on motherboard
acpi0: <PTLTD   RSDT> on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pci0: <unknown> at device 0.1 (no driver attached)
pci0: <base peripheral> at device 1.0 (no driver attached)
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
em0: <Intel(R) PRO/1000 Network Connection Version - 3.2.18> port
0x2000-0x203f mem 0xd4200000-0xd421ffff irq 30 at device 3.0 on pci2
em0: Ethernet address: 00:30:48:2f:51:d2
em1: <Intel(R) PRO/1000 Network Connection Version - 3.2.18> port
0x2040-0x207f mem 0xd4220000-0xd423ffff irq 31 at device 3.1 on pci2
em1: Ethernet address: 00:30:48:2f:51:d3
pci1: <base peripheral, interrupt controller> at device 0.1 (no driver
attached)
pcib3: <ACPI PCI-PCI bridge> at device 0.2 on pci1
pci3: <ACPI PCI bus> on pcib3
aac0: <Adaptec SCSI RAID 2025ZCR> mem
0xd5000000-0xd5ffffff,0xd4400000-0xd45fffff,0xdc000000-0xdfffffff irq 58 at
device 3.0 on pci3
aac0: Enabling 64-bit address support
aac0: New comm. interface enabled
aac0: Adaptec Raid Controller 2.0.0-1
aacp0: <SCSI Passthrough Bus> on aac0
aacp1: <SCSI Passthrough Bus> on aac0
pci1: <base peripheral, interrupt controller> at device 0.3 (no driver
attached)
pcib4: <ACPI PCI-PCI bridge> irq 16 at device 4.0 on pci0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> irq 16 at device 6.0 on pci0
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci6: <ACPI PCI bus> on pcib6
pci6: <display, VGA> at device 1.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH5 UDMA100 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1420-0x142f at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 flags 0x10 on acpi0
uart1: console (19200,n,8,1)
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FAST]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
orm0: <ISA Option ROMs> at iomem
0xc0000-0xc7fff,0xc8000-0xc8fff,0xc9000-0xcd7ff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
module_register_init: MOD_LOAD (amr_linux, 0xffffffff804e02b0, 0) error 6
ipfw2 (+ipv6) initialized, divert loadable, rule-based forwarding disabled,
default to accept, logging limited to 100 packets/entry by default
acd0: CDROM <CD-224E-N/1.AA> at ata1-master UDMA33
aacd0: <RAID 1 (Mirror)> on aac0
aacd0: 17451MB (35739904 sectors)
aacd1: <RAID 1 (Mirror)> on aac0
aacd1: 139981MB (286681856 sectors)
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!
Trying to mount root from ufs:/dev/aacd0s1a

Thanks,
Matt