Some ZFS experience

Wed Feb 6 16:24:32 UTC 2008

Hi list monkeys!

I've been using ZFS on freebsd since August 2007. 
It was running on 4x 9GB SCSI drives in raidz1 configuration powered by
some old bx133 motherboard with 2x CPUs on 350MHz with 512MB of RAM.
Applications running on it were couple of jails spinning low traffic
mail servers and mysqls. All was running "smooth" considering the amount
of fancy hardware :)
However about week or so I upgraded that box by placing all 4 drives in
new machine with P4 grade CPU and 1GB of ram, *but* I failed to connect
all 4 drives in new enviroment with 1 scsi cable (was too short to reach
4th drive) and decided to give it a shot and see what happens with 1 hdd
offline (omg i was expecting that will work).

What happened:
1. Base system running on gmirror volume consisting of 4 mirrors booted
up normally in degraded mode with 3 of 4 drives online.
2. /data powered by raidz1 zfs was showing 3 drives, 2 online and 1
faulted. zfs list was showing that volume is unavailable due lack of
spares.

I'm not sure which drive I didnt connect, was it da0, da1, da2 or da3 so
i tried one more combination by cabling 3 drives in little different
fashion. This time 2 of 3 drives showed as faulted.
I've left it and got extra cabling the next day. Connected all 4 drives
and ZFS pool continued working normally like it was in previous box.

What was wrong when 3 drives were connected? Wasnt supposed to raidz1
survive lack of 1 drive? Or I did something wrong there?

I didnt do 'zpool export' prior to physical migration of the drives. 
I'm using FreeBSD 7.0-PRERELEASE pulled from cvsup before 2 weeks. No
custom patches.

conf files and dmesg follows:

cat /boot/loader.conf 
geom_mirror_load="YES"
zfs_load="YES"
vfs.zfs.zil_disable="1"
vfs.zfs.prefetch_disable="1"
#vm.kmem_size="335708160"
vm.kmem_size="512M"
vm.kmem_size_max="512M"
hw.ata.wc=0

FreeBSD anarki.default.co.yu 7.0-PRERELEASE FreeBSD 7.0-PRERELEASE #3:
Thu Jan 31 13:27:25 CET 2008

dmesg:
Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights
reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.0-PRERELEASE #3: Thu Jan 31 13:27:25 CET 2008
    root at anarki.default.co.yu:/usr/obj/usr/src/sys/ANARKI
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Celeron(R) D CPU 3.06GHz (3081.50-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0xf64  Stepping = 4

Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0xe41d<SSE3,RSVD2,MON,DS_CPL,CNXT-ID,CX16,xTPR,PDCM>
  AMD Features=0x20100000<NX,LM>
  AMD Features2=0x1<LAHF>
real memory  = 1065287680 (1015 MB)
avail memory = 1028866048 (981 MB)
ACPI APIC Table: <GBT    GBTUACPI>
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <GBT GBTUACPI> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, 3f6f0000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on
acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
cpu0: <ACPI CPU> on acpi0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <PCI-PCI bridge> irq 16 at device 1.0 on pci0
pci1: <PCI bus> on pcib1
vgapci0: <VGA-compatible display> port 0xd000-0xd007 mem
0xe2000000-0xe207ffff,0xd0000000-0xdfffffff,0xe2080000-0xe20bffff irq 16
at device 2.0 on pci0
agp0: <Intel 82945G (945G GMCH) SVGA controller> on vgapci0
agp0: detected 7932k stolen memory
agp0: aperture size is 256M
pcib2: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci2: <ACPI PCI bus> on pcib2
ahc0: <Adaptec 29160 Ultra160 SCSI adapter> port 0xb000-0xb0ff mem
0xe1000000-0xe1000fff irq 18 at device 2.0 on pci2
ahc0: [ITHREAD]
aic7892: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs
re0: <RealTek 8169SC/8110SC Single-chip Gigabit Ethernet> port
0xb400-0xb4ff mem 0xe1001000-0xe10010ff irq 21 at device 5.0 on pci2
miibus0: <MII bus> on re0
rgephy0: <RTL8169S/8110S/8211B media interface> PHY 1 on miibus0
rgephy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
1000baseT-FDX, auto
re0: Ethernet address: 00:1d:7d:32:fc:ab
re0: [FILTER]
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on
acpi0
fdc0: [FILTER]
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on
acpi0
sio0: type 16550A
sio0: [FILTER]
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
pmtimer0 on isa0
orm0: <ISA Option ROM> at iomem 0xc0000-0xca7ff pnpid ORM0000 on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on
isa0
ata0 at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0
ata0: [ITHREAD]
ata1 at port 0x170-0x177,0x376 irq 15 on isa0
ata1: [ITHREAD]
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
WARNING: ZFS is considered to be an experimental feature in FreeBSD.
Timecounter "TSC" frequency 3081499911 Hz quality 800
Timecounters tick every 1.000 msec
Waiting 5 seconds for SCSI devices to settle
ZFS filesystem version 6
ZFS storage pool version 6
da0 at ahc0 bus 0 target 1 lun 0
da0: <IBM-PSG DPSS-318350N  F S96E> Fixed Direct Access SCSI-3 device 
da0: 40.000MB/s transfers (20.000MHz, offset 63, 16bit)
da0: Command Queueing Enabled
da0: 17357MB (35548320 512 byte sectors: 255H 63S/T 2212C)
da1 at ahc0 bus 0 target 2 lun 0
da1: <IBM DGHS18V 03E0> Fixed Direct Access SCSI-3 device 
da1: 40.000MB/s transfers (20.000MHz, offset 15, 16bit)
da1: Command Queueing Enabled
da1: 17501MB (35843670 512 byte sectors: 255H 63S/T 2231C)
da2 at ahc0 bus 0 target 4 lun 0
da2: <IBM DGHS18U 0350> Fixed Direct Access SCSI-3 device 
da2: 40.000MB/s transfers (20.000MHz, offset 15, 16bit)
da2: Command Queueing Enabled
da2: 17501MB (35843670 512 byte sectors: 255H 63S/T 2231C)
da3 at ahc0 bus 0 target 12 lun 0
da3: <IBM-PSG DPSS-318350N  F S96E> Fixed Direct Access SCSI-3 device 
da3: 40.000MB/s transfers (20.000MHz, offset 63, 16bit)
da3: Command Queueing Enabled
da3: 17357MB (35548320 512 byte sectors: 255H 63S/T 2212C)
GEOM_MIRROR: Device mirror/sys1s1 launched (4/4).
Trying to mount root from ufs:/dev/mirror/sys1s1a