zpool degraded - 'UNAVAIL cannot open' functioning drive

Vye Wilson vyeperman at gmail.com
Thu Aug 7 04:35:14 UTC 2008 

Hello,

I setup a raidz1 zpool to test ZFS with a device failure and to see how
quickly the zpool could be resilvered. The system I'm using has a backplane
that all the drives are connected to, so everything is hotswappable. I
created the raidz1 zpool and then removed one of the drives. zpool status
showed that the pool was degraded but online. Ok great, so lets bring the
now functioning drive back online.

[root at Touzyoh /home/vye]# zpool online ztemp ad18
Bringing device ad18 online

Everything looks good... lets check the zpool status

  pool: ztemp
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist
for
    the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-D3
 scrub: resilver completed with 0 errors on Wed Aug  6 20:59:54 2008
config:

    NAME        STATE     READ WRITE CKSUM
    ztemp       DEGRADED     0     0     0
      raidz1    DEGRADED     0     0     0
        ad10    ONLINE       0     0     0
        ad14    ONLINE       0     0     0
        ad18    UNAVAIL      0     0     0  cannot open

errors: No known data errors

Doh! still degraded. It shows 'UNAVAIL cannot open' I've tried rebooting but
it will not open that drive at all. According to dmesg the drive is
functional, and if I destroy the pool and recreate it the drive works fine.
I wasn't able to find any similar issues on this mailing list or in google.
Does anyone have any ideas? I've attached my dmesg output.

Thanks.
-- 
--Vye
-------------- next part --------------
Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.0-STABLE #1: Wed Jul 30 10:29:43 PDT 2008
    vye at Touzyoh.wow.com:/usr/obj/usr/src/sys/GENERIC
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Opteron(tm) Processor 248 (2200.01-MHz K8-class CPU)
  Origin = "AuthenticAMD"  Id = 0xf5a  Stepping = 10
  Features=0x78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2>
  AMD Features=0xe0500800<SYSCALL,NX,MMX+,LM,3DNow!+,3DNow!>
usable memory = 3744727040 (3571 MB)
avail memory  = 3621384192 (3453 MB)
ACPI APIC Table: <A M I  OEMAPIC >
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
ioapic0 <Version 1.1> irqs 0-23 on motherboard
ioapic1 <Version 1.1> irqs 24-47 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
acpi0: <A M I OEMRSDT> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 100000, dff00000 (3) failed
acpi0: reservation of dfefd000, 400 (3) failed
acpi0: reservation of dfefe000, 400 (3) failed
acpi0: reservation of dfeff000, 1000 (3) failed
acpi0: reservation of 0, a0000 (3) failed
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pci0: <memory> at device 0.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
pci0: <serial bus, SMBus> at device 1.1 (no driver attached)
ohci0: <OHCI (generic) USB controller> mem 0xfdbfc000-0xfdbfcfff irq 20 at device 2.0 on pci0
ohci0: [GIANT-LOCKED]
ohci0: [ITHREAD]
usb0: OHCI version 1.0, legacy support
usb0: SMM does not respond, resetting
usb0: <OHCI (generic) USB controller> on ohci0
usb0: USB revision 1.0
uhub0: <nVidia OHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 10 ports with 10 removable, self powered
ehci0: <NVIDIA nForce4 USB 2.0 controller> mem 0xfdbe0000-0xfdbe00ff irq 21 at device 2.1 on pci0
ehci0: [GIANT-LOCKED]
ehci0: [ITHREAD]
usb1: EHCI version 1.0
usb1: companion controller, 4 ports each: usb0
usb1: <NVIDIA nForce4 USB 2.0 controller> on ehci0
usb1: USB revision 2.0
uhub1: <nVidia EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb1
uhub1: 10 ports with 10 removable, self powered
pci0: <multimedia, audio> at device 4.0 (no driver attached)
atapci0: <nVidia nForce CK804 UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x3000-0x300f at device 6.0 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
atapci1: <nVidia nForce CK804 SATA300 controller> port 0xcc00-0xcc07,0xc800-0xc803,0xc400-0xc407,0xc000-0xc003,0xbc00-0xbc0f mem 0xfdbff000-0xfdbfffff irq 20 at device 7.0 on pci0
atapci1: [ITHREAD]
ata2: <ATA channel 0> on atapci1
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci1
ata3: [ITHREAD]
atapci2: <nVidia nForce CK804 SATA300 controller> port 0xb800-0xb807,0xb400-0xb403,0xb000-0xb007,0xac00-0xac03,0xa800-0xa80f mem 0xfdbfe000-0xfdbfefff irq 21 at device 8.0 on pci0
atapci2: [ITHREAD]
ata4: <ATA channel 0> on atapci2
ata4: [ITHREAD]
ata5: <ATA channel 1> on atapci2
ata5: [ITHREAD]
pcib1: <ACPI PCI-PCI bridge> at device 9.0 on pci0
pci5: <ACPI PCI bus> on pcib1
vgapci0: <VGA-compatible display> port 0x9800-0x98ff mem 0xfc000000-0xfcffffff,0xfdaff000-0xfdafffff irq 16 at device 4.0 on pci5
pcib2: <ACPI PCI-PCI bridge> at device 11.0 on pci0
pci4: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 12.0 on pci0
pci2: <ACPI PCI bus> on pcib3
bge0: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x4101> mem 0xfb8f0000-0xfb8fffff irq 17 at device 0.0 on pci2
miibus0: <MII bus> on bge0
brgphy0: <BCM5750 10/100/1000baseTX PHY> PHY 1 on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bge0: Ethernet address: 00:30:48:57:5a:de
bge0: [ITHREAD]
pcib4: <ACPI PCI-PCI bridge> at device 13.0 on pci0
pci3: <ACPI PCI bus> on pcib4
bge1: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x4101> mem 0xfb9f0000-0xfb9fffff irq 18 at device 0.0 on pci3
miibus1: <MII bus> on bge1
brgphy1: <BCM5750 10/100/1000baseTX PHY> PHY 1 on miibus1
brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bge1: Ethernet address: 00:30:48:57:5a:df
bge1: [ITHREAD]
pcib5: <ACPI PCI-PCI bridge> at device 14.0 on pci0
pci1: <ACPI PCI bus> on pcib5
pcib6: <ACPI Host-PCI bridge> on acpi0
pci128: <ACPI PCI bus> on pcib6
pci128: <memory> at device 0.0 (no driver attached)
pci128: <memory> at device 1.0 (no driver attached)
atapci3: <nVidia nForce CK804 SATA300 controller> port 0xfc00-0xfc07,0xf800-0xf803,0xf400-0xf407,0xf000-0xf003,0xec00-0xec0f mem 0xfdeff000-0xfdefffff irq 44 at device 7.0 on pci128
atapci3: [ITHREAD]
ata6: <ATA channel 0> on atapci3
ata6: [ITHREAD]
ata7: <ATA channel 1> on atapci3
ata7: [ITHREAD]
atapci4: <nVidia nForce CK804 SATA300 controller> port 0xe800-0xe807,0xe400-0xe403,0xe000-0xe007,0xdc00-0xdc03,0xd800-0xd80f mem 0xfdefe000-0xfdefefff irq 45 at device 8.0 on pci128
atapci4: [ITHREAD]
ata8: <ATA channel 0> on atapci4
ata8: [ITHREAD]
ata9: <ATA channel 1> on atapci4
ata9: [ITHREAD]
pcib7: <ACPI PCI-PCI bridge> at device 13.0 on pci128
pci132: <ACPI PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> at device 14.0 on pci128
pci129: <ACPI PCI bus> on pcib8
pcib9: <PCI-PCI bridge> at device 0.0 on pci129
pci131: <PCI bus> on pcib9
arcmsr0: <Areca SATA Host Adapter RAID Controller (RAID6 capable)
> mem 0xfddff000-0xfddfffff,0xfe000000-0xfe3fffff irq 40 at device 14.0 on pci131
ARECA RAID ADAPTER0: Driver Version 1.20.00.15 2007-10-07 
ARECA RAID ADAPTER0: FIRMWARE VERSION V1.41 2006-5-24  
arcmsr0: [ITHREAD]
pcib10: <PCI-PCI bridge> at device 0.2 on pci129
pci130: <PCI bus> on pcib10
cpu0: <ACPI CPU> on acpi0
powernow0: <Cool`n'Quiet K8> on cpu0
device_attach: powernow0 attach returned 6
cpu1: <ACPI CPU> on acpi0
powernow1: <Cool`n'Quiet K8> on cpu1
device_attach: powernow1 attach returned 6
acpi_button0: <Power Button> on acpi0
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A
sio0: [FILTER]
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
sio1: [FILTER]
fdc0: <floppy drive controller (FDE)> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FILTER]
ppc0: <Parallel port> port 0x378-0x37f irq 7 on acpi0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppbus0: <Parallel port bus> on ppc0
ppbus0: [ITHREAD]
plip0: <PLIP network interface> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
ppc0: [GIANT-LOCKED]
ppc0: [ITHREAD]
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc97ff,0xc9800-0xcafff,0xcb000-0xcbfff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
ad6: 152627MB <Seagate ST3160811AS 3.AAE> at ata3-master SATA150
ad10: 953869MB <SAMSUNG HD103UJ 1AA01109> at ata5-master SATA300
ad14: 286168MB <Seagate ST3300620AS 3.AAE> at ata7-master SATA150
ad18: 286168MB <Seagate ST3300620AS 3.AAE> at ata9-master SATA150
Waiting 5 seconds for SCSI devices to settle
(probe16:arcmsr0:0:16:0): inquiry data fails comparison at DV1 step
da0 at arcmsr0 bus 0 target 0 lun 0
da0: <Areca ARC-1260-VOL#00 R001> Fixed Direct Access SCSI-3 device 
da0: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit)
da0: 286102MB (585937408 512 byte sectors: 255H 63S/T 36472C)
da1 at arcmsr0 bus 0 target 0 lun 1
da1: <Areca ARC-1260-VOL#01 R001> Fixed Direct Access SCSI-3 device 
da1: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit)
da1: 953674MB (1953124352 512 byte sectors: 255H 63S/T 121576C)
da2 at arcmsr0 bus 0 target 0 lun 2
da2: <Areca ARC-1260-VOL#02 R001> Fixed Direct Access SCSI-3 device 
da2: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit)
da2: 572204MB (1171874816 512 byte sectors: 255H 63S/T 72945C)
SMP: AP CPU #1 Launched!
Trying to mount root from ufs:/dev/da0s1a
WARNING: ZFS is considered to be an experimental feature in FreeBSD.
ZFS filesystem version 6
ZFS storage pool version 6

More information about the freebsd-fs mailing list