Hang on reboot with ZIL on SSD
bsd at todoo.biz
bsd at todoo.biz
Tue Feb 26 18:30:18 UTC 2013
Hi,
I have a quite big server that I am tuning with FreeNAS running on It.
It is based on an Intel Server and uses an Adaptec Controler ASR-6805 for a potential 12 disks pool (only 6 deployed for the moment).
I have two more SSD disks intended for the ZIL cache. Connected directly on the mother board.
It comes equipped with 32Gb of memory ECC.
The system is installed on a specific dongle on the mother board (4Gb SLC dongle).
The system is happy (= reboots without stopping at the real end of the reboot) as long as there is no SSD involved for the ZIL. As soon as the SSD are running, system freezes (or at least can't proceed with the reboot).
It really freezes at the real end of the reboot after :
> Syncing disks, vnodes remaining…*0 0 0 0 done
> All buffers synced.
> Uptime: 3d4h12min
I have to manually Power-Cycle the unit for It to complete the reboot.
Here is the output of the dmesg :
> [root at freenas] ~# dmesg
> Copyright (c) 1992-2012 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
> The Regents of the University of California. All rights reserved.
> FreeBSD is a registered trademark of The FreeBSD Foundation.
> FreeBSD 8.3-RELEASE-p5 #2 r244158M: Wed Dec 12 10:04:42 PST 2012
> root at build.ixsystems.com:/home/jpaetzel/8.3.0/os-base/amd64/usr/home/jpaetzel/8.3.0/FreeBSD/src/sys/FREENAS.amd64 amd64
> Timecounter "i8254" frequency 1193182 Hz quality 0
> CPU: Intel(R) Xeon(R) CPU E5-2609 0 @ 2.40GHz (2394.25-MHz K8-class CPU)
> Origin = "GenuineIntel" Id = 0x206d7 Family = 6 Model = 2d Stepping = 7
> Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
> Features2=0x17bee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,AVX>
> AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
> AMD Features2=0x1<LAHF>
> TSC: P-state invariant
> real memory = 34359738368 (32768 MB)
> avail memory = 33071357952 (31539 MB)
> ACPI APIC Table: <INTEL S2600GZ>
> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
> FreeBSD/SMP: 1 package(s) x 4 core(s)
> cpu0 (BSP): APIC ID: 0
> cpu1 (AP): APIC ID: 2
> cpu2 (AP): APIC ID: 4
> cpu3 (AP): APIC ID: 6
> WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
> ACPI Warning: Invalid length for Pm1aControlBlock: 32, using default 16 (20101013/tbfadt-707)
> ioapic0 <Version 2.0> irqs 0-23 on motherboard
> ioapic1 <Version 2.0> irqs 24-47 on motherboard
> kbd1 at kbdmux0
> hpt27xx: RocketRAID 27xx controller driver v1.0 (Dec 12 2012 10:04:31)
> cryptosoft0: <software crypto> on motherboard
> aesni0: <AES-CBC,AES-XTS> on motherboard
> acpi0: <INTEL S2600GZ> on motherboard
> acpi0: [ITHREAD]
> acpi0: Power Button (fixed)
> acpi0: reservation of 0, 9d000 (3) failed
> Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
> acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
> cpu0: <ACPI CPU> on acpi0
> cpu1: <ACPI CPU> on acpi0
> cpu2: <ACPI CPU> on acpi0
> cpu3: <ACPI CPU> on acpi0
> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
> pci0: <ACPI PCI bus> on pcib0
> pcib1: <ACPI PCI-PCI bridge> irq 47 at device 1.0 on pci0
> pci1: <ACPI PCI bus> on pcib1
> pcib2: <ACPI PCI-PCI bridge> irq 47 at device 1.1 on pci0
> pci2: <ACPI PCI bus> on pcib2
> igb0: <Intel(R) PRO/1000 Network Connection version - 2.3.1> port 0x1060-0x107f mem 0xd2160000-0xd217ffff,0xd21b0000-0xd21b3fff irq 27 at device 0.0 on pci2
> igb0: Using MSIX interrupts with 5 vectors
> igb0: Ethernet address: 00:1e:67:54:9f:cd
> igb0: [ITHREAD]
> igb0: [ITHREAD]
> igb0: [ITHREAD]
> igb0: [ITHREAD]
> igb0: [ITHREAD]
> igb1: <Intel(R) PRO/1000 Network Connection version - 2.3.1> port 0x1040-0x105f mem 0xd2140000-0xd215ffff,0xd21a0000-0xd21a3fff irq 30 at device 0.1 on pci2
> igb1: Using MSIX interrupts with 5 vectors
> igb1: Ethernet address: 00:1e:67:54:9f:ce
> igb1: [ITHREAD]
> igb1: [ITHREAD]
> igb1: [ITHREAD]
> igb1: [ITHREAD]
> igb1: [ITHREAD]
> igb2: <Intel(R) PRO/1000 Network Connection version - 2.3.1> port 0x1020-0x103f mem 0xd2120000-0xd213ffff,0xd2190000-0xd2193fff irq 28 at device 0.2 on pci2
> igb2: Using MSIX interrupts with 5 vectors
> igb2: Ethernet address: 00:1e:67:54:9f:cf
> igb2: [ITHREAD]
> igb2: [ITHREAD]
> igb2: [ITHREAD]
> igb2: [ITHREAD]
> igb2: [ITHREAD]
> igb3: <Intel(R) PRO/1000 Network Connection version - 2.3.1> port 0x1000-0x101f mem 0xd2100000-0xd211ffff,0xd2180000-0xd2183fff irq 29 at device 0.3 on pci2
> igb3: Using MSIX interrupts with 5 vectors
> igb3: Ethernet address: 00:1e:67:54:9f:d0
> igb3: [ITHREAD]
> igb3: [ITHREAD]
> igb3: [ITHREAD]
> igb3: [ITHREAD]
> igb3: [ITHREAD]
> pcib3: <ACPI PCI-PCI bridge> irq 47 at device 2.0 on pci0
> pci4: <ACPI PCI bus> on pcib3
> pcib4: <ACPI PCI-PCI bridge> irq 47 at device 2.2 on pci0
> pci5: <ACPI PCI bus> on pcib4
> pcib5: <ACPI PCI-PCI bridge> irq 16 at device 3.0 on pci0
> pci6: <ACPI PCI bus> on pcib5
> aacu0: <Adaptec RAID Controller> mem 0xd1c00000-0xd1ffffff,0xd2050000-0xd20507ff,0xd2040000-0xd20400ff irq 40 at device 0.0 on pci6
> aacu0: Enable Raw I/O
> aacu0: Enable 64-bit array
> aacu0: New comm. interface type1 enabled
> aacu0: [ITHREAD]
> aacu0: Adaptec 6805, aac driver 3.1.2-30035
> aacp0: <Container Bus> on aacu0
> aacp1: <SCSI Passthrough Bus> on aacu0
> aacp2: <SCSI Passthrough Bus> on aacu0
> aacp3: <SCSI Passthrough Bus> on aacu0
> pcib6: <ACPI PCI-PCI bridge> irq 16 at device 3.2 on pci0
> pci7: <ACPI PCI bus> on pcib6
> pci0: <base peripheral> at device 4.0 (no driver attached)
> pci0: <base peripheral> at device 4.1 (no driver attached)
> pci0: <base peripheral> at device 4.2 (no driver attached)
> pci0: <base peripheral> at device 4.3 (no driver attached)
> pci0: <base peripheral> at device 4.4 (no driver attached)
> pci0: <base peripheral> at device 4.5 (no driver attached)
> pci0: <base peripheral> at device 4.6 (no driver attached)
> pci0: <base peripheral> at device 4.7 (no driver attached)
> pci0: <base peripheral> at device 5.0 (no driver attached)
> pci0: <base peripheral> at device 5.2 (no driver attached)
> pcib7: <ACPI PCI-PCI bridge> irq 16 at device 17.0 on pci0
> pci8: <ACPI PCI bus> on pcib7
> pci0: <simple comms> at device 22.0 (no driver attached)
> pci0: <simple comms> at device 22.1 (no driver attached)
> ehci0: <EHCI (generic) USB 2.0 controller> mem 0xd2320000-0xd23203ff irq 22 at device 26.0 on pci0
> ehci0: [ITHREAD]
> usbus0: EHCI version 1.0
> usbus0: <EHCI (generic) USB 2.0 controller> on ehci0
> pcib8: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
> pci9: <ACPI PCI bus> on pcib8
> pcib9: <ACPI PCI-PCI bridge> irq 19 at device 28.7 on pci0
> pci10: <ACPI PCI bus> on pcib9
> vgapci0: <VGA-compatible display> mem 0xd0000000-0xd0ffffff,0xd1810000-0xd1813fff,0xd1000000-0xd17fffff irq 19 at device 0.0 on pci10
> ehci1: <EHCI (generic) USB 2.0 controller> mem 0xd2310000-0xd23103ff irq 20 at device 29.0 on pci0
> ehci1: [ITHREAD]
> usbus1: EHCI version 1.0
> usbus1: <EHCI (generic) USB 2.0 controller> on ehci1
> pcib10: <ACPI PCI-PCI bridge> at device 30.0 on pci0
> pci11: <ACPI PCI bus> on pcib10
> isab0: <PCI-ISA bridge> at device 31.0 on pci0
> isa0: <ISA bus> on isab0
> ahci0: <Intel Patsburg AHCI SATA controller> port 0x2070-0x2077,0x2060-0x2063,0x2050-0x2057,0x2040-0x2043,0x2020-0x203f mem 0xd2300000-0xd23007ff irq 21 at device 31.2 on pci0
> ahci0: [ITHREAD]
> ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
> ahcich0: <AHCI channel> at channel 0 on ahci0
> ahcich0: [ITHREAD]
> ahcich1: <AHCI channel> at channel 1 on ahci0
> ahcich1: [ITHREAD]
> pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
> pcib11: <ACPI Host-PCI bridge> on acpi0
> pci255: <ACPI PCI bus> on pcib11
> pci255: <base peripheral> at device 8.0 (no driver attached)
> pci255: <base peripheral> at device 9.0 (no driver attached)
> pci255: <base peripheral> at device 10.0 (no driver attached)
> pci255: <base peripheral> at device 10.1 (no driver attached)
> pci255: <base peripheral> at device 10.2 (no driver attached)
> pci255: <base peripheral> at device 10.3 (no driver attached)
> pci255: <base peripheral> at device 11.0 (no driver attached)
> pci255: <base peripheral> at device 11.3 (no driver attached)
> pci255: <base peripheral> at device 12.0 (no driver attached)
> pci255: <base peripheral> at device 12.1 (no driver attached)
> pci255: <base peripheral> at device 12.6 (no driver attached)
> pci255: <base peripheral> at device 12.7 (no driver attached)
> pci255: <base peripheral> at device 13.0 (no driver attached)
> pci255: <base peripheral> at device 13.1 (no driver attached)
> pci255: <base peripheral> at device 13.6 (no driver attached)
> pci255: <base peripheral> at device 14.0 (no driver attached)
> pci255: <dasp> at device 14.1 (no driver attached)
> pci255: <base peripheral> at device 15.0 (no driver attached)
> pci255: <base peripheral> at device 15.1 (no driver attached)
> pci255: <base peripheral> at device 15.2 (no driver attached)
> pci255: <base peripheral> at device 15.3 (no driver attached)
> pci255: <base peripheral> at device 15.4 (no driver attached)
> pci255: <base peripheral> at device 15.5 (no driver attached)
> pci255: <base peripheral> at device 15.6 (no driver attached)
> pci255: <base peripheral> at device 16.0 (no driver attached)
> pci255: <base peripheral> at device 16.1 (no driver attached)
> pci255: <base peripheral> at device 16.2 (no driver attached)
> pci255: <base peripheral> at device 16.3 (no driver attached)
> pci255: <base peripheral> at device 16.4 (no driver attached)
> pci255: <base peripheral> at device 16.5 (no driver attached)
> pci255: <base peripheral> at device 16.6 (no driver attached)
> pci255: <base peripheral> at device 16.7 (no driver attached)
> pci255: <base peripheral> at device 17.0 (no driver attached)
> pci255: <base peripheral> at device 19.0 (no driver attached)
> pci255: <dasp> at device 19.1 (no driver attached)
> pci255: <dasp> at device 19.4 (no driver attached)
> pci255: <dasp> at device 19.5 (no driver attached)
> pci255: <base peripheral> at device 19.6 (no driver attached)
> acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
> Timecounter "HPET" frequency 14318180 Hz quality 900
> atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
> atrtc0: Warning: Couldn't map I/O.
> uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
> uart0: [FILTER]
> uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
> uart1: [FILTER]
> orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff,0xc9000-0xc9fff,0xca000-0xcafff,0xcb000-0xcbfff,0xcc000-0xd27ff on isa0
> sc0: <System console> at flags 0x100 on isa0
> sc0: VGA <16 virtual consoles, flags=0x300>
> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
> atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
> atkbd0: <AT Keyboard> irq 1 on atkbdc0
> kbd0 at atkbd0
> atkbd0: [GIANT-LOCKED]
> atkbd0: [ITHREAD]
> ppc0: cannot reserve I/O port range
> coretemp0: <CPU On-Die Thermal Sensors> on cpu0
> est0: <Enhanced SpeedStep Frequency Control> on cpu0
> p4tcc0: <CPU Frequency Thermal Control> on cpu0
> coretemp1: <CPU On-Die Thermal Sensors> on cpu1
> est1: <Enhanced SpeedStep Frequency Control> on cpu1
> est: CPU supports Enhanced Speedstep, but is not recognized.
> est: cpu_vendor GenuineIntel, msr 1ee700001800
> device_attach: est1 attach returned 6
> p4tcc1: <CPU Frequency Thermal Control> on cpu1
> coretemp2: <CPU On-Die Thermal Sensors> on cpu2
> est2: <Enhanced SpeedStep Frequency Control> on cpu2
> p4tcc2: <CPU Frequency Thermal Control> on cpu2
> coretemp3: <CPU On-Die Thermal Sensors> on cpu3
> est3: <Enhanced SpeedStep Frequency Control> on cpu3
> est: CPU supports Enhanced Speedstep, but is not recognized.
> est: cpu_vendor GenuineIntel, msr 1f1000001800
> device_attach: est3 attach returned 6
> p4tcc3: <CPU Frequency Thermal Control> on cpu3
> fuse4bsd: version 0.3.9-pre1, FUSE ABI 7.8
> Timecounters tick every 1.000 msec
> hpt27xx: no controller detected.
> usbus0: 480Mbps High Speed USB v2.0
> usbus1: 480Mbps High Speed USB v2.0
> ugen0.1: <Intel> at usbus0
> uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
> ugen1.1: <Intel> at usbus1
> uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
> uhub0: 2 ports with 2 removable, self powered
> uhub1: 2 ports with 2 removable, self powered
> ugen0.2: <vendor 0x8087> at usbus0
> uhub2: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus0
> ugen1.2: <vendor 0x8087> at usbus1
> uhub3: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus1
> uhub2: 6 ports with 6 removable, self powered
> uhub3: 8 ports with 8 removable, self powered
> ugen1.3: <InnoDisk> at usbus1
> umass0: <InnoDisk USB EDC, class 0/0, rev 2.00/9.10, addr 3> on usbus1
> ugen1.4: <vendor 0x04d9> at usbus1
> ukbd0: <vendor 0x04d9 USB Keyboard, class 0/0, rev 1.10/3.10, addr 4> on usbus1
> kbd2 at ukbd0
> uhid0: <vendor 0x04d9 USB Keyboard, class 0/0, rev 1.10/3.10, addr 4> on usbus1
> ugen1.5: <American Megatrends Inc.> at usbus1
> ukbd1: <Keyboard Interface> on usbus1
> kbd3 at ukbd1
> ums0: <Mouse Interface> on usbus1
> ums0: 3 buttons and [Z] coordinates ID=0
> ada0 at ahcich0 bus 0 scbus4 target 0 lun 0
> ada0: <INTEL SSDSA2BZ100G3 6PB10362> ATA-8 SATA 2.x device
> ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
> ada0: Command Queueing enabled
> ada0: 95396MB (195371568 512 byte sectors: 16H 63S/T 16383C)
> ada1 at ahcich1 bus 0 scbus5 target 0 lun 0
> ada1: <INTEL SSDSA2BZ100G3 6PB10362> ATA-8 SATA 2.x device
> ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
> ada1: Command Queueing enabled
> ada1: 95396MB (195371568 512 byte sectors: 16H 63S/T 16383C)
> da6 at umass-sim0 bus 0 scbus6 target 0 lun 0
> da6: <InnoDisk USB EDC 0910> Fixed Direct Access SCSI-0 device
> da6: 40.000MB/s transfers
> da6: 3920MB (8028160 512 byte sectors: 255H 63S/T 499C)
> SMP: AP CPU #1 Launched!
> SMP: AP CPU #2 Launched!
> SMP: AP CPU #3 Launched!
> da0 at aacp1 bus 0 scbus1 target 0 lun 0
> da0: <WDC WD30EFRX-68AX9N0 80.0> Fixed Direct Access SCSI-5 device
> da0: 300.000MB/s transfers
> da0: Command Queueing enabled
> da0: 2856950MB (5851033600 512 byte sectors: 255H 63S/T 364209C)
> da1 at aacp1 bus 0 scbus1 target 1 lun 0
> da1: <WDC WD30EFRX-68AX9N0 80.0> Fixed Direct Access SCSI-5 device
> da1: 300.000MB/s transfers
> da1: Command Queueing enabled
> da1: 2856950MB (5851033600 512 byte sectors: 255H 63S/T 364209C)
> da2 at aacp1 bus 0 scbus1 target 4 lun 0
> da2: <WDC WD30EFRX-68AX9N0 80.0> Fixed Direct Access SCSI-5 device
> da2: 300.000MB/s transfers
> da2: Command Queueing enabled
> da2: 2856950MB (5851033600 512 byte sectors: 255H 63S/T 364209C)
> da3 at aacp1 bus 0 scbus1 target 5 lun 0
> da3: <WDC WD30EFRX-68AX9N0 80.0> Fixed Direct Access SCSI-5 device
> da3: 300.000MB/s transfers
> da3: Command Queueing enabled
> da3: 2856950MB (5851033600 512 byte sectors: 255H 63S/T 364209C)
> da4 at aacp1 bus 0 scbus1 target 6 lun 0
> da4: <WDC WD30EFRX-68AX9N0 80.0> Fixed Direct Access SCSI-5 device
> da4: 300.000MB/s transfers
> da4: Command Queueing enabled
> da4: 2856950MB (5851033600 512 byte sectors: 255H 63S/T 364209C)
> da5 at aacp1 bus 0 scbus1 target 7 lun 0
> da5: <WDC WD30EFRX-68AX9N0 80.0> Fixed Direct Access SCSI-5 device
> da5: 300.000MB/s transfers
> da5: Command Queueing enabled
> da5: 2856950MB (5851033600 512 byte sectors: 255H 63S/T 364209C)
> GEOM: da6s1: geometry does not match label (16h,63s != 255h,63s).
> Trying to mount root from ufs:/dev/ufs/FreeNASs1a
> ZFS filesystem version 5
> ZFS storage pool version 28
Here is an example of the pool with only one disk used for ZIL (not the target) - But I tried this in order to see how It worked…
> [root at freenas] ~# zpool status
> pool: tank
> state: ONLINE
> scan: none requested
> config:
>
> NAME STATE READ WRITE CKSUM
> tank ONLINE 0 0 0
> raidz2-0 ONLINE 0 0 0
> gptid/053bf90c-803f-11e2-a8db-001e67549fcd ONLINE 0 0 0
> gptid/08a7dcbd-803f-11e2-a8db-001e67549fcd ONLINE 0 0 0
> gptid/0c0767d0-803f-11e2-a8db-001e67549fcd ONLINE 0 0 0
> gptid/0f6ac358-803f-11e2-a8db-001e67549fcd ONLINE 0 0 0
> gptid/12ce2b68-803f-11e2-a8db-001e67549fcd ONLINE 0 0 0
> gptid/162f71c0-803f-11e2-a8db-001e67549fcd ONLINE 0 0 0
> logs
> gptid/17980c47-803f-11e2-a8db-001e67549fcd ONLINE 0 0 0
>
> errors: No known data errors
Is there some tuning to be done on the Bios when you use disks as ZIL ?
Thanks for your feedback.
G.B.
________________________________________________
«?»¥«?»§«?»¥«?»§«?»¥«?»§«?»¥«?»§«?»¥«?»§«?»¥«?»§
¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
Your provider of OpenSource Appliances
www.osnet.eu
________________________________________________
«?»¥«?»§«?»¥«?»§«?»¥«?»§«?»¥«?»§«?»¥«?»§«?»¥«?»§
¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
PGP ID --> 0x1BA3C2FD
More information about the freebsd-questions
mailing list