kernel panic twice in two days. how to tell if it is hardware or software?

[Vick Khera]

Two days in a row last week, one of my FreeBSD/amd64 systems running 10.0
failed with the following panic:

[panic: vm_radix_insert: key 46f is already present  cpuid = 9  KDB: stack
backtrace:  db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
0xfffffe3fcf1b1820  ]
[kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe3fcf1b18d0  panic() at
panic+0x155/frame 0xfffffe3fcf1b1950  ]
[vm_radix_insert() at vm_radix_insert+0x2ed/frame 0xfffffe3fcf1b19b0
 vm_page_cache() at vm_page_cache+0x121/frame 0xfffffe3fcf1b19f0  ]
[vm_pageout() at vm_pageout+0x8f7/frame 0xfffffe3fcf1b1a70  fork_exit() at
fork_exit+0x9a/frame 0xfffffe3fcf1b1ab0  ]
[fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe3fcf1b1ab0  ---
trap 0, rip = 0, rsp = 0xfffffe3fcf1b1b70, rbp = 0 ---  ]

At this point the serial console was unresponsive, network unresponsive.
After about 35 minutes, I just power cycled it. I figured if it was dumping
the RAM I should let it finish, but it didn't seem to be doing so. The
above was captured by my serial port terminal server. Obviously nothing
made it to the log files on the server itself. After this panic, I disabled
crash dumps on a hunch that it may have been stuck doing that.

Almost exactly 25 hours later, the same panic occurred:

[panic: vm_radix_insert: key 46f is already present  cpuid = 0  KDB: stack
backtrace:  db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
0xfffffe3fcf1b1820  ]
[kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe3fcf1b18d0  panic() at
panic+0x155/frame 0xfffffe3fcf1b1950  ]
[vm_radix_insert() at vm_radix_insert+0x2ed/frame 0xfffffe3fcf1b19b0
 vm_page_cache() at vm_page_cache+0x121/frame 0xfffffe3fcf1b19f0  ]
[vm_pageout() at vm_pageout+0x8f7/frame 0xfffffe3fcf1b1a70  fork_exit() at
fork_exit+0x9a/frame 0xfffffe3fcf1b1ab0  ]
[fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe3fcf1b1ab0  ---
trap 0, rip = 0, rsp = 0xfffffe3fcf1b1b70, rbp = 0 ---  ]

Again, the serial console was unresponsive. Since I knew there was not
going to be a crash dump, I remote power-cyced it within a couple of
minutes and it recovered quite nicely. I *LOVE* ZFS. :)

This machine has 256GB of ECC RAM, and 2 6-core Intel Xeon CPUs with
hyper-threading turned off.  The uptime was 300 days when it crashed first
on Thursday. The sole purpose of this machine is running Postgres 9.2
server. The other things it does is run NRPE for nagios monitoring and
slony1 for database replication.

The boot messages are below. This is running FreeBSD 10.0. I wanted to try
the tips in Section 17.6 of the handbook, but the panic did not emit an
instruction pointer for me. I run a custom kernel.

I have plans to update this system to FreeBSD 10.1, but I wanted to see if
there was any other debugging I could do here to figure out if it is a
kernel bug or if it is a hardware fault starting to happen. The system in
question is just about 1 year old.


The rest of this email is just the dmesg.boot:

Copyright (c) 1992-2014 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.0-RELEASE #1 r261210: Tue Jan 28 10:36:57 EST 2014
    vivek at devbox.int.kcilink.com:/n/yertle1/sources/usr10/obj/n/yertle1/sources/usr10/src/sys/KCI64
amd64
FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610
CPU: Intel(R) Xeon(R) CPU E5-2630L v2 @ 2.40GHz (2400.05-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x306e4  Family = 0x6  Model = 0x3e
 Stepping = 4

Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>

Features2=0x7fbee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  Standard Extended Features=0x281<GSFSBASE,SMEP,ENHMOVSB>
  TSC: P-state invariant, performance statistics
real memory  = 274894684160 (262160 MB)
avail memory = 267173515264 (254796 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: < >
FreeBSD/SMP: Multiprocessor System Detected: 12 CPUs
FreeBSD/SMP: 2 package(s) x 6 core(s)
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  2
 cpu2 (AP): APIC ID:  4
 cpu3 (AP): APIC ID:  6
 cpu4 (AP): APIC ID:  8
 cpu5 (AP): APIC ID: 10
 cpu6 (AP): APIC ID: 32
 cpu7 (AP): APIC ID: 34
 cpu8 (AP): APIC ID: 36
 cpu9 (AP): APIC ID: 38
 cpu10 (AP): APIC ID: 40
 cpu11 (AP): APIC ID: 42
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
ioapic2 <Version 2.0> irqs 48-71 on motherboard
random: <Software, Yarrow> initialized
kbd1 at kbdmux0
acpi0: <SUPERM SMCI--MB> on motherboard
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
cpu4: <ACPI CPU> on acpi0
cpu5: <ACPI CPU> on acpi0
cpu6: <ACPI CPU> on acpi0
cpu7: <ACPI CPU> on acpi0
cpu8: <ACPI CPU> on acpi0
cpu9: <ACPI CPU> on acpi0
cpu10: <ACPI CPU> on acpi0
cpu11: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 350
Event timer "HPET1" frequency 14318180 Hz quality 340
Event timer "HPET2" frequency 14318180 Hz quality 340
Event timer "HPET3" frequency 14318180 Hz quality 340
Event timer "HPET4" frequency 14318180 Hz quality 340
Event timer "HPET5" frequency 14318180 Hz quality 340
Event timer "HPET6" frequency 14318180 Hz quality 340
Event timer "HPET7" frequency 14318180 Hz quality 340
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 26 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> mem 0xdfc00000-0xdfc03fff irq 16 at device 0.0
on pci1
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 8.0 on pci2
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> irq 26 at device 1.1 on pci0
pci4: <ACPI PCI bus> on pcib4
igb0: <Intel(R) PRO/1000 Network Connection version - 2.4.0> port
0x8020-0x803f mem 0xdfb20000-0xdfb3ffff,0xdfb44000-0xdfb47fff irq 27 at
device 0.0 on pci4
igb0: Using MSIX interrupts with 9 vectors
igb0: Ethernet address: 00:25:90:c3:be:14
igb0: Bound queue 0 to cpu 0
igb0: Bound queue 1 to cpu 1
igb0: Bound queue 2 to cpu 2
igb0: Bound queue 3 to cpu 3
igb0: Bound queue 4 to cpu 4
igb0: Bound queue 5 to cpu 5
igb0: Bound queue 6 to cpu 6
igb0: Bound queue 7 to cpu 7
igb1: <Intel(R) PRO/1000 Network Connection version - 2.4.0> port
0x8000-0x801f mem 0xdfb00000-0xdfb1ffff,0xdfb40000-0xdfb43fff irq 30 at
device 0.1 on pci4
igb1: Using MSIX interrupts with 9 vectors
igb1: Ethernet address: 00:25:90:c3:be:15
igb1: Bound queue 0 to cpu 8
igb1: Bound queue 1 to cpu 9
igb1: Bound queue 2 to cpu 10
igb1: Bound queue 3 to cpu 11
igb1: Bound queue 4 to cpu 0
igb1: Bound queue 5 to cpu 1
igb1: Bound queue 6 to cpu 2
igb1: Bound queue 7 to cpu 3
pcib5: <ACPI PCI-PCI bridge> irq 33 at device 2.0 on pci0
pci6: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> irq 41 at device 3.0 on pci0
pci7: <ACPI PCI bus> on pcib6
mps0: <LSI SAS2308> port 0x7000-0x70ff mem
0xdfa40000-0xdfa4ffff,0xdfa00000-0xdfa3ffff irq 40 at device 0.0 on pci7
mps0: Firmware: 17.00.01.00, Driver: 16.00.00.00-fbsd
mps0: IOCCapabilities:
1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
pci0: <base peripheral> at device 4.0 (no driver attached)
pci0: <base peripheral> at device 4.1 (no driver attached)
pci0: <base peripheral> at device 4.2 (no driver attached)
pci0: <base peripheral> at device 4.3 (no driver attached)
pci0: <base peripheral> at device 4.4 (no driver attached)
pci0: <base peripheral> at device 4.5 (no driver attached)
pci0: <base peripheral> at device 4.6 (no driver attached)
pci0: <base peripheral> at device 4.7 (no driver attached)
pci0: <base peripheral> at device 5.0 (no driver attached)
pci0: <base peripheral> at device 5.2 (no driver attached)
pci0: <simple comms> at device 22.0 (no driver attached)
pci0: <simple comms> at device 22.1 (no driver attached)
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xdfd01000-0xdfd013ff irq 16
at device 26.0 on pci0
usbus0: EHCI version 1.0
usbus0 on ehci0
pcib7: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
pci8: <ACPI PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> irq 19 at device 28.7 on pci0
pci9: <ACPI PCI bus> on pcib8
pcib9: <PCI-PCI bridge> at device 0.0 on pci9
pci10: <PCI bus> on pcib9
pcib10: <PCI-PCI bridge> at device 0.0 on pci10
pci11: <PCI bus> on pcib10
pcib11: <PCI-PCI bridge> at device 0.0 on pci11
pci12: <PCI bus> on pcib11
vgapci0: <VGA-compatible display> mem
0xde000000-0xdeffffff,0xdf800000-0xdf803fff,0xdf000000-0xdf7fffff irq 19 at
device 0.0 on pci12
vgapci0: Boot video device
pcib12: <PCI-PCI bridge> at device 1.0 on pci10
pci13: <PCI bus> on pcib12
ehci1: <EHCI (generic) USB 2.0 controller> mem 0xdfd00000-0xdfd003ff irq 23
at device 29.0 on pci0
usbus1: EHCI version 1.0
usbus1 on ehci1
pcib13: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci14: <ACPI PCI bus> on pcib13
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
pci0: <dasp> at device 31.6 (no driver attached)
pcib14: <ACPI Host-PCI bridge> on acpi0
pci127: <ACPI PCI bus> on pcib14
pci127: <base peripheral> at device 8.0 (no driver attached)
pci127: <base peripheral> at device 9.0 (no driver attached)
pci127: <base peripheral> at device 10.0 (no driver attached)
pci127: <base peripheral> at device 10.1 (no driver attached)
pci127: <base peripheral> at device 10.2 (no driver attached)
pci127: <base peripheral> at device 10.3 (no driver attached)
pci127: <base peripheral> at device 11.0 (no driver attached)
pci127: <base peripheral> at device 11.3 (no driver attached)
pci127: <base peripheral> at device 12.0 (no driver attached)
pci127: <base peripheral> at device 12.1 (no driver attached)
pci127: <base peripheral> at device 12.2 (no driver attached)
pci127: <base peripheral> at device 13.0 (no driver attached)
pci127: <base peripheral> at device 13.1 (no driver attached)
pci127: <base peripheral> at device 13.2 (no driver attached)
pci127: <base peripheral> at device 14.0 (no driver attached)
pci127: <dasp> at device 14.1 (no driver attached)
pci127: <base peripheral> at device 15.0 (no driver attached)
pci127: <base peripheral> at device 15.1 (no driver attached)
pci127: <base peripheral> at device 15.2 (no driver attached)
pci127: <base peripheral> at device 15.3 (no driver attached)
pci127: <base peripheral> at device 15.4 (no driver attached)
pci127: <base peripheral> at device 15.5 (no driver attached)
pci127: <base peripheral> at device 16.0 (no driver attached)
pci127: <base peripheral> at device 16.1 (no driver attached)
pci127: <base peripheral> at device 16.2 (no driver attached)
pci127: <base peripheral> at device 16.3 (no driver attached)
pci127: <base peripheral> at device 16.4 (no driver attached)
pci127: <base peripheral> at device 16.5 (no driver attached)
pci127: <base peripheral> at device 16.6 (no driver attached)
pci127: <base peripheral> at device 16.7 (no driver attached)
pci127: <base peripheral> at device 19.0 (no driver attached)
pci127: <dasp> at device 19.1 (no driver attached)
pci127: <base peripheral> at device 19.4 (no driver attached)
pci127: <dasp> at device 19.5 (no driver attached)
pci127: <base peripheral> at device 22.0 (no driver attached)
pci127: <base peripheral> at device 22.1 (no driver attached)
pci127: <base peripheral> at device 22.2 (no driver attached)
pcib15: <ACPI Host-PCI bridge> on acpi0
pci128: <ACPI PCI bus> on pcib15
pcib16: <ACPI PCI-PCI bridge> irq 50 at device 1.0 on pci128
pci129: <ACPI PCI bus> on pcib16
pcib17: <ACPI PCI-PCI bridge> irq 57 at device 2.0 on pci128
pci130: <ACPI PCI bus> on pcib17
pci128: <base peripheral> at device 4.0 (no driver attached)
pci128: <base peripheral> at device 4.1 (no driver attached)
pci128: <base peripheral> at device 4.2 (no driver attached)
pci128: <base peripheral> at device 4.3 (no driver attached)
pci128: <base peripheral> at device 4.4 (no driver attached)
pci128: <base peripheral> at device 4.5 (no driver attached)
pci128: <base peripheral> at device 4.6 (no driver attached)
pci128: <base peripheral> at device 4.7 (no driver attached)
pci128: <base peripheral> at device 5.0 (no driver attached)
pci128: <base peripheral> at device 5.2 (no driver attached)
pcib18: <ACPI Host-PCI bridge> on acpi0
pci255: <ACPI PCI bus> on pcib18
pci255: <base peripheral> at device 8.0 (no driver attached)
pci255: <base peripheral> at device 9.0 (no driver attached)
pci255: <base peripheral> at device 10.0 (no driver attached)
pci255: <base peripheral> at device 10.1 (no driver attached)
pci255: <base peripheral> at device 10.2 (no driver attached)
pci255: <base peripheral> at device 10.3 (no driver attached)
pci255: <base peripheral> at device 11.0 (no driver attached)
pci255: <base peripheral> at device 11.3 (no driver attached)
pci255: <base peripheral> at device 12.0 (no driver attached)
pci255: <base peripheral> at device 12.1 (no driver attached)
pci255: <base peripheral> at device 12.2 (no driver attached)
pci255: <base peripheral> at device 13.0 (no driver attached)
pci255: <base peripheral> at device 13.1 (no driver attached)
pci255: <base peripheral> at device 13.2 (no driver attached)
pci255: <base peripheral> at device 14.0 (no driver attached)
pci255: <dasp> at device 14.1 (no driver attached)
pci255: <base peripheral> at device 15.0 (no driver attached)
pci255: <base peripheral> at device 15.1 (no driver attached)
pci255: <base peripheral> at device 15.2 (no driver attached)
pci255: <base peripheral> at device 15.3 (no driver attached)
pci255: <base peripheral> at device 15.4 (no driver attached)
pci255: <base peripheral> at device 15.5 (no driver attached)
pci255: <base peripheral> at device 16.0 (no driver attached)
pci255: <base peripheral> at device 16.1 (no driver attached)
pci255: <base peripheral> at device 16.2 (no driver attached)
pci255: <base peripheral> at device 16.3 (no driver attached)
pci255: <base peripheral> at device 16.4 (no driver attached)
pci255: <base peripheral> at device 16.5 (no driver attached)
pci255: <base peripheral> at device 16.6 (no driver attached)
pci255: <base peripheral> at device 16.7 (no driver attached)
pci255: <base peripheral> at device 19.0 (no driver attached)
pci255: <dasp> at device 19.1 (no driver attached)
pci255: <base peripheral> at device 19.4 (no driver attached)
pci255: <dasp> at device 19.5 (no driver attached)
pci255: <base peripheral> at device 22.0 (no driver attached)
pci255: <base peripheral> at device 22.1 (no driver attached)
pci255: <base peripheral> at device 22.2 (no driver attached)
acpi_button0: <Power Button> on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (115200,n,8,1)
ipmi0: <IPMI System Interface> port 0xca2,0xca3 on acpi0
ipmi0: KCS mode found at io 0xca2 on acpi
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xce000-0xcefff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
acpi_throttle1: <ACPI CPU Throttling> on cpu1
acpi_throttle1: failed to attach P_CNT
device_attach: acpi_throttle1 attach returned 6
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
acpi_throttle2: <ACPI CPU Throttling> on cpu2
acpi_throttle2: failed to attach P_CNT
device_attach: acpi_throttle2 attach returned 6
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
acpi_throttle3: <ACPI CPU Throttling> on cpu3
acpi_throttle3: failed to attach P_CNT
device_attach: acpi_throttle3 attach returned 6
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
acpi_throttle4: <ACPI CPU Throttling> on cpu4
acpi_throttle4: failed to attach P_CNT
device_attach: acpi_throttle4 attach returned 6
coretemp4: <CPU On-Die Thermal Sensors> on cpu4
acpi_throttle5: <ACPI CPU Throttling> on cpu5
acpi_throttle5: failed to attach P_CNT
device_attach: acpi_throttle5 attach returned 6
coretemp5: <CPU On-Die Thermal Sensors> on cpu5
acpi_throttle6: <ACPI CPU Throttling> on cpu6
acpi_throttle6: failed to attach P_CNT
device_attach: acpi_throttle6 attach returned 6
coretemp6: <CPU On-Die Thermal Sensors> on cpu6
acpi_throttle7: <ACPI CPU Throttling> on cpu7
acpi_throttle7: failed to attach P_CNT
device_attach: acpi_throttle7 attach returned 6
coretemp7: <CPU On-Die Thermal Sensors> on cpu7
acpi_throttle8: <ACPI CPU Throttling> on cpu8
acpi_throttle8: failed to attach P_CNT
device_attach: acpi_throttle8 attach returned 6
coretemp8: <CPU On-Die Thermal Sensors> on cpu8
acpi_throttle9: <ACPI CPU Throttling> on cpu9
acpi_throttle9: failed to attach P_CNT
device_attach: acpi_throttle9 attach returned 6
coretemp9: <CPU On-Die Thermal Sensors> on cpu9
acpi_throttle10: <ACPI CPU Throttling> on cpu10
acpi_throttle10: failed to attach P_CNT
device_attach: acpi_throttle10 attach returned 6
coretemp10: <CPU On-Die Thermal Sensors> on cpu10
acpi_throttle11: <ACPI CPU Throttling> on cpu11
acpi_throttle11: failed to attach P_CNT
device_attach: acpi_throttle11 attach returned 6
coretemp11: <CPU On-Die Thermal Sensors> on cpu11
Timecounters tick every 1.000 msec
random: unblocking device.
usbus0: 480Mbps High Speed USB v2.0
usbus1: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
ipmi0: IPMI device rev. 1, firmware rev. 2.19, version 2.0
ipmi0: Number of channels 3
ipmi0: Attached watchdog
da0 at mps0 bus 0 scbus0 target 0 lun 0
da0: <ATA INTEL SSDSC2BA10 0265> Fixed Direct Access SCSI-6 device
da0: Serial Number BTTV335200Z3100FGN
da0: 600.000MB/s transfers
da0: Command Queueing enabled
da0: 95396MB (195371568 512 byte sectors: 255H 63S/T 12161C)
da1 at mps0 bus 0 scbus0 target 1 lun 0
da1: <ATA INTEL SSDSC2BA10 0265> Fixed Direct Access SCSI-6 device
da1: Serial Number BTTV335106X6100FGN
da1: 600.000MB/s transfers
da1: Command Queueing enabled
da1: 95396MB (195371568 512 byte sectors: 255H 63S/T 12161C)
da2 at mps0 bus 0 scbus0 target 4 lun 0
da2: <SEAGATE ST600MM0026 0001> Fixed Direct Access SCSI-6 device
da2: Serial Number Z0M0032V00009328Z4K4
da2: 600.000MB/s transfers
da2: Command Queueing enabled
da2: 572325MB (1172123568 512 byte sectors: 255H 63S/T 72961C)
da3 at mps0 bus 0 scbus0 target 5 lun 0
da3: <SEAGATE ST600MM0026 0001> Fixed Direct Access SCSI-6 device
da3: Serial Number Z0M0006B00009329D7CW
da3: 600.000MB/s transfers
da3: Command Queueing enabled
da3: 572325MB (1172123568 512 byte sectors: 255H 63S/T 72961C)
da4 at mps0 bus 0 scbus0 target 6 lun 0
da4: <SEAGATE ST600MM0026 0001> Fixed Direct Access SCSI-6 device
da4: Serial Number Z0M002MT00009329D793
da4: 600.000MB/s transfers
da4: Command Queueing enabled
da4: 572325MB (1172123568 512 byte sectors: 255H 63S/T 72961C)
da5 at mps0 bus 0 scbus0 target 7 lun 0
da5: <SEAGATE ST600MM0026 0001> Fixed Direct Access SCSI-6 device
da5: Serial Number Z0M003L200009338Y1P0
da5: 600.000MB/s transfers
da5: Command Queueing enabled
da5: 572325MB (1172123568 512 byte sectors: 255H 63S/T 72961C)
da6 at mps0 bus 0 scbus0 target 8 lun 0
da6: <ATA INTEL SSDSC2BB08 0355> Fixed Direct Access SCSI-6 device
da6: Serial Number BTWL33900172080KGN
da6: 600.000MB/s transfers
da6: Command Queueing enabled
da6: 76319MB (156301488 512 byte sectors: 255H 63S/T 9729C)
da7 at mps0 bus 0 scbus0 target 9 lun 0
da7: <ATA INTEL SSDSC2BB08 0355> Fixed Direct Access SCSI-6 device
da7: Serial Number BTWL339000A5080KGN
da7: 600.000MB/s transfers
da7: Command Queueing enabled
da7: 76319MB (156301488 512 byte sectors: 255H 63S/T 9729C)
GEOM_MIRROR: Device mirror/gmroot launched (2/2).
GEOM_MIRROR: Device mirror/gmusr launched (2/2).
GEOM_MIRROR: Device mirror/gmvar launched (1/2).
GEOM_MIRROR: Device gmvar: rebuilding provider da6p5.
SMP: AP CPU #10 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #11 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #9 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #8 Launched!
Timecounter "TSC-low" frequency 1200025156 Hz quality 1000
Root mount waiting for: usbus1 usbus0
uhub1: 2 ports with 2 removable, self powered
uhub0: 2 ports with 2 removable, self powered
Root mount waiting for: usbus1 usbus0
ugen0.2: <vendor 0x8087> at usbus0
uhub2: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on
usbus0
ugen1.2: <vendor 0x8087> at usbus1
uhub3: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on
usbus1
Root mount waiting for: usbus1 usbus0
uhub2: 6 ports with 6 removable, self powered
uhub3: 8 ports with 8 removable, self powered
ugen0.3: <American Megatrends Inc.> at usbus0
ukbd0: <Keyboard Interface> on usbus0
kbd2 at ukbd0
Trying to mount root from ufs:/dev/mirror/gmroot [rw]...
WARNING: / was not properly dismounted
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
igb0: link state changed to UP