Problems with em0 failing to initialize on stable?

[gnn at freebsd.org]

Hi,

I am running 7-STABLE with machines that are net booted.  On occasion,
that is not with any level of predictability, this happens:

em0: Hardware Initialization Failed
em0: Unable to initialize the hardware

which of course stops the machine in its tracks.  A normal dmesg is
also included.

Any steps I should take to help debug this?

Best,
George


Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.0-STABLE #2: Wed Apr 16 22:18:10 EDT 2008
    gnn at builder:/builder/gnn/FreeBSD.7/src/sys/amd64/compile/HRT_FBSD7_DEBUG
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU           X5365  @ 3.00GHz (3000.12-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x6fb  Stepping = 11
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x4e3bd<SSE3,RSVD2,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA>
  AMD Features=0x20100800<SYSCALL,NX,LM>
  AMD Features2=0x1<LAHF>
  Cores per package: 4
usable memory = 8580186112 (8182 MB)
avail memory  = 8293068800 (7908 MB)
ACPI APIC Table: <PTLTD          APIC  >
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  2
 cpu3 (AP): APIC ID:  3
 cpu4 (AP): APIC ID:  4
 cpu5 (AP): APIC ID:  5
 cpu6 (AP): APIC ID:  6
 cpu7 (AP): APIC ID:  7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
acpi0: <PTLTD   RSDT> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
cpu1: <ACPI CPU> on acpi0
acpi_throttle1: <ACPI CPU Throttling> on cpu1
acpi_throttle1: failed to attach P_CNT
device_attach: acpi_throttle1 attach returned 6
cpu2: <ACPI CPU> on acpi0
acpi_throttle2: <ACPI CPU Throttling> on cpu2
acpi_throttle2: failed to attach P_CNT
device_attach: acpi_throttle2 attach returned 6
cpu3: <ACPI CPU> on acpi0
acpi_throttle3: <ACPI CPU Throttling> on cpu3
acpi_throttle3: failed to attach P_CNT
device_attach: acpi_throttle3 attach returned 6
cpu4: <ACPI CPU> on acpi0
acpi_throttle4: <ACPI CPU Throttling> on cpu4
acpi_throttle4: failed to attach P_CNT
device_attach: acpi_throttle4 attach returned 6
cpu5: <ACPI CPU> on acpi0
acpi_throttle5: <ACPI CPU Throttling> on cpu5
acpi_throttle5: failed to attach P_CNT
device_attach: acpi_throttle5 attach returned 6
cpu6: <ACPI CPU> on acpi0
acpi_throttle6: <ACPI CPU Throttling> on cpu6
acpi_throttle6: failed to attach P_CNT
device_attach: acpi_throttle6 attach returned 6
cpu7: <ACPI CPU> on acpi0
acpi_throttle7: <ACPI CPU Throttling> on cpu7
acpi_throttle7: failed to attach P_CNT
device_attach: acpi_throttle7 attach returned 6
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci2
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> irq 18 at device 2.0 on pci2
pci4: <ACPI PCI bus> on pcib4
em0: <Intel(R) PRO/1000 Network Connection Version - 6.7.3> port
0x2000-0x201f mem 0xdaa20000-0xdaa3ffff,0xdaa00000-0xdaa1ffff irq 18
at device 0.0 on pci4
em0: Using MSI interrupt
em0: Ethernet address: 00:30:48:7d:7c:56
em0: [FILTER]
em1: <Intel(R) PRO/1000 Network Connection Version - 6.7.3> port 0x2020-0x203f mem 0xdaa60000-0xdaa7ffff,0xdaa40000-0xdaa5ffff irq 19 at device 0.1 on pci4
em1: Using MSI interrupt
em1: Ethernet address: 00:30:48:7d:7c:57
em1: [FILTER]
pcib5: <ACPI PCI-PCI bridge> at device 0.3 on pci1
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci6: <ACPI PCI bus> on pcib6
cxgbc0: <Chelsio T310 RNIC, 1 port> mem 0xda801000-0xda801fff,0xda000000-0xda7fffff,0xda800000-0xda800fff irq 16 at device 0.0 on pci6
cxgb0: <Port 0 10GBASE-SR> on cxgbc0
cxgb0: Ethernet address: 00:07:43:05:26:16
pcib7: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci7: <ACPI PCI bus> on pcib7
3ware device driver for 9000 series storage controllers, version: 3.70.05.001
twa0: <3ware 9000 series Storage Controller> port 0x3000-0x30ff mem 0xd8000000-0xd9ffffff,0xdab00000-0xdab00fff irq 18 at device 0.0 on pci7
twa0: [ITHREAD]
twa0: INFO: (0x15: 0x1300): Controller details:: Model 9650SE-4LPML, 4 ports, Firmware FE9X 3.08.00.016, BIOS BE9X 3.08.00.004
pci0: <base peripheral> at device 8.0 (no driver attached)
uhci0: <Intel 631XESB/632XESB/3100 USB controller USB-1> port 0x1800-0x181f irq 17 at device 29.0 on pci0
uhci0: [GIANT-LOCKED]
uhci0: [ITHREAD]
usb0: <Intel 631XESB/632XESB/3100 USB controller USB-1> on uhci0
usb0: USB revision 1.0
uhub0: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 2 ports with 2 removable, self powered
uhci1: <Intel 631XESB/632XESB/3100 USB controller USB-2> port 0x1820-0x183f irq 19 at device 29.1 on pci0
uhci1: [GIANT-LOCKED]
uhci1: [ITHREAD]
usb1: <Intel 631XESB/632XESB/3100 USB controller USB-2> on uhci1
usb1: USB revision 1.0
uhub1: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
uhub1: 2 ports with 2 removable, self powered
uhci2: <Intel 631XESB/632XESB/3100 USB controller USB-3> port 0x1840-0x185f irq 18 at device 29.2 on pci0
uhci2: [GIANT-LOCKED]
uhci2: [ITHREAD]
usb2: <Intel 631XESB/632XESB/3100 USB controller USB-3> on uhci2
usb2: USB revision 1.0
uhub2: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2
uhub2: 2 ports with 2 removable, self powered
ehci0: <Intel 63XXESB USB 2.0 controller> mem 0xdaf00000-0xdaf003ff irq 17 at device 29.7 on pci0
ehci0: [GIANT-LOCKED]
ehci0: [ITHREAD]
usb3: EHCI version 1.0
usb3: companion controllers, 2 ports each: usb0 usb1 usb2
usb3: <Intel 63XXESB USB 2.0 controller> on ehci0
usb3: USB revision 2.0
uhub3: <Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb3
uhub3: 6 ports with 6 removable, self powered
pcib8: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci8: <ACPI PCI bus> on pcib8
vgapci0: <VGA-compatible display> port 0x4000-0x40ff mem 0xd0000000-0xd7ffffff,0xdac00000-0xdac0ffff irq 18 at device 1.0 on pci8
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel 63XXESB2 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1860-0x186f at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A, console
sio0: [FILTER]
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
sio1: [FILTER]
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FILTER]
orm0: <ISA Option ROMs> at iomem 0xc0000-0xcafff,0xcb000-0xcbfff,0xcc000-0xcdfff on isa0
ppc0: cannot reserve I/O port range
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x100>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
acd0: CDRW <TSSTcorpCDW/DVD SN-M242D/SB01> at ata0-slave UDMA33
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #6 Launched!
da0 at twa0 bus 0 target 0 lun 0
da0: <AMCC 9650SE-4LP DISK 3.08> Fixed Direct Access SCSI-5 device 
da0: 100.000MB/s transfers
da0: 1430481MB (2929625088 512 byte sectors: 255H 63S/T 182360C)
Trying to mount root from nfs:
NFS ROOT: 10.0.8.100:/builder/testlab1
cxgbc0: [ITHREAD]
setting up interrupt for port=0
cxgbc0: [ITHREAD]
bind qsets
cxgbc0: enabling interrupts on port=0