Multiple USB drives stability question

Sat Aug 15 15:03:15 UTC 2009

I am now trying to rsync large files from the 320GB gmirror+gjournal device to the 2nd 1TB gmirror+gjournal device.  Using gstat I see the 320GB device active all the time while the 1TB device loads in spurts.  There will be periods of multiple seconds where the target providers are completely idle while the source providers are still reporting 100% active.

Is there any tuning I should be investigating for these GEOM classes?

--- On Fri, 8/14/09, Jeff Richards <bsd24x7 at yahoo.com> wrote:

From: Jeff Richards <bsd24x7 at yahoo.com>
Subject: Re: Multiple USB drives stability question
To: freebsd-stable at freebsd.org
Date: Friday, August 14, 2009, 11:04 PM

I just tested my 2nd 1TB gmirror device on another system with FBSD 7.2.  I was getting full throughput on the drive and no lockup using bonnie++ and also monitoring with gstat.

I then moved those drives back on my main server.  When I booted the system I hung on the 320GB gmirror devices.  Previously the 1st 1TB gmirror and 320GB gmirror were attached to the integrated USB ports on the motherboard.  I moved the 320GB gmirror to a PCI USB adapter.

The 2 320GB drives in the gmirror were da5 and da6.  Here's what I saw on the console:

(da6:umass-sim6:6:0:0): SYNCHRONIZE CACHE(10). CDB: 35 0 0 0 0 0 0 0 0 0
(da6:umass-sim6:6:0:0): CAM Status: SCSI Status Error
(da6:umass-sim6:6:0:0): SCSI Status: Check Condition
(da6:umass-sim6:6:0:0): ILLEGAL REQUEST asc:20,0
(da6:umass-sim6:6:0:0): Invalid command operation mode
(da6:umass-sim6:6:0:0): Unretryable error
GEOM_MIRROR: Request failed (error=5), da6[READ(offset=512, length=512)]
GEOM_MIRROR: Device gm-san: provider da6 disconnected.
(da5:umass-sim5:5:0:0): SYNCHRONIZE CACHE(10). CDB: 35 0 0 0 0 0 0 0 0 0
(da5:umass-sim5:5:0:0): CAM Status: SCSI Status Error
(da5:umass-sim5:5:0:0): SCSI Status: Check Condition
(da5:umass-sim5:5:0:0): ILLEGAL REQUEST asc:20,0
(da5:umass-sim5:5:0:0): Invalid command operation mode
(da5:umass-sim5:5:0:0): Unretryable error
GEOM_JOURNAL: BIO_FLUSH not supported by mirror/gm-san.

I waited for a few minutes with no change in the console.  I then detached one of the USB drives (which happened to be da6) and saw this:

umass6: at uhub7 port 4 (addr 4) disconnected
(da6:umass-sim6:6:0:0): lost device

Nothing else changed for a few minutes so I powered off the system.  When I brought it back up the 320GB gmirror device was out of sync, but apart from that all devices were online.

Below are the kernel messages from the second boot:

Copyright (c) 1992-2009 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.2-RELEASE #0: Fri May  1 08:49:13 UTC 2009
    root at walker.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Celeron(R) CPU 2.26GHz (2266.67-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0xf49  Stepping = 9
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x441d<SSE3,DTES64,MON,DS_CPL,CNXT-ID,xTPR>
  AMD Features2=0x1<LAHF>
real memory  = 1877868544 (1790 MB)
avail memory = 1826934784 (1742 MB)
ACPI APIC Table: <P4M900 AWRDACPI>
ioapic0 <Version 0.3> irqs 0-23 on motherboard
ioapic1 <Version 0.3> irqs 24-47 on motherboard
kbd1 at kbdmux0
acpi0: <P4M900 AWRDACPI> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, 6fde0000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
acpi_hpet0: <High Precision Event Timer> iomem 0xfe800000-0xfe8003ff on acpi0
device_attach: acpi_hpet0 attach returned 12
acpi_button0: <Power Button> on acpi0
acpi_button1: <Sleep Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <PCI-PCI bridge> at device 1.0 on pci0
pci1: <PCI bus> on pcib1
vgapci0: <VGA-compatible display> mem 0xc0000000-0xcfffffff,0xfb000000-0xfbffffff irq 16 at device 0.0 on pci1
pcib2: <ACPI PCI-PCI bridge> irq 27 at device 2.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 31 at device 3.0 on pci0
pci3: <ACPI PCI bus> on pcib3
atapci0: <VIA 8237S SATA150 controller> port 0xfc00-0xfc07,0xf800-0xf803,0xf400-0xf407,0xf000-0xf003,0xec00-0xec0f,0xe800-0xe8ff irq 21 at device 15.0 on pci0
atapci0: [ITHREAD]
ata2: <ATA channel 0> on atapci0
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci0
ata3: [ITHREAD]
atapci1: <VIA 8237S UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xe400-0xe40f at device 15.1 on pci0
ata0: <ATA channel 0> on atapci1
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci1
ata1: [ITHREAD]
uhci0: <VIA 83C572 USB controller> port 0xe000-0xe01f irq 20 at device 16.0 on pci0
uhci0: [GIANT-LOCKED]
uhci0: [ITHREAD]
usb0: <VIA 83C572 USB controller> on uhci0
usb0: USB revision 1.0
uhub0: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 2 ports with 2 removable, self powered
uhci1: <VIA 83C572 USB controller> port 0xdc00-0xdc1f irq 22 at device 16.1 on pci0
uhci1: [GIANT-LOCKED]
uhci1: [ITHREAD]
usb1: <VIA 83C572 USB controller> on uhci1
usb1: USB revision 1.0
uhub1: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
uhub1: 2 ports with 2 removable, self powered
uhci2: <VIA 83C572 USB controller> port 0xd800-0xd81f irq 21 at device 16.2 on pci0
uhci2: [GIANT-LOCKED]
uhci2: [ITHREAD]
usb2: <VIA 83C572 USB controller> on uhci2
usb2: USB revision 1.0
uhub2: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2
uhub2: 2 ports with 2 removable, self powered
uhci3: <VIA 83C572 USB controller> port 0xd400-0xd41f irq 23 at device 16.3 on pci0
uhci3: [GIANT-LOCKED]
uhci3: [ITHREAD]
usb3: <VIA 83C572 USB controller> on uhci3
usb3: USB revision 1.0
uhub3: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb3
uhub3: 2 ports with 2 removable, self powered
ehci0: <VIA VT6202 USB 2.0 controller> mem 0xfdfff000-0xfdfff0ff irq 21 at device 16.4 on pci0
ehci0: [GIANT-LOCKED]
ehci0: [ITHREAD]
usb4: EHCI version 1.0
usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3
usb4: <VIA VT6202 USB 2.0 controller> on ehci0
usb4: USB revision 2.0
uhub4: <VIA EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb4
uhub4: 8 ports with 8 removable, self powered
umass0: <Initio HDT721010SLA360, class 0/0, rev 2.00/1.43, addr 2> on uhub4
umass1: <Initio WD1002FBYS-01A6B, class 0/0, rev 2.00/1.43, addr 3> on uhub4
umass2: <Initio WD1002FBYS-01A6B, class 0/0, rev 2.00/1.43, addr 4> on uhub4
umass3: <Initio HDT721010SLA360, class 0/0, rev 2.00/1.43, addr 5> on uhub4
isab0: <PCI-ISA bridge> at device 17.0 on pci0
isa0: <ISA bus> on isab0
vr0: <VIA VT6102 Rhine II 10/100BaseTX> port 0xd000-0xd0ff mem 0xfdffe000-0xfdffe0ff irq 23 at device 18.0 on pci0
vr0: Quirks: 0x0
vr0: Revision: 0x7c
miibus0: <MII bus> on vr0
rlphy0: <RTL8201L 10/100 media interface> PHY 1 on miibus0
rlphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
vr0: Ethernet address: 00:e0:4d:2b:19:ed
vr0: [ITHREAD]
pcib4: <ACPI PCI-PCI bridge> at device 19.1 on pci0
pci4: <ACPI PCI bus> on pcib4
ohci0: <NEC uPD 9210 USB controller> mem 0xfdaff000-0xfdafffff irq 16 at device 3.0 on pci4
ohci0: [GIANT-LOCKED]
ohci0: [ITHREAD]
usb5: OHCI version 1.0
usb5: <NEC uPD 9210 USB controller> on ohci0
usb5: USB revision 1.0
uhub5: <NEC OHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb5
uhub5: 3 ports with 3 removable, self powered
ohci1: <NEC uPD 9210 USB controller> mem 0xfdafe000-0xfdafefff irq 17 at device 3.1 on pci4
ohci1: [GIANT-LOCKED]
ohci1: [ITHREAD]
usb6: OHCI version 1.0
usb6: <NEC uPD 9210 USB controller> on ohci1
usb6: USB revision 1.0
uhub6: <NEC OHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb6
uhub6: 2 ports with 2 removable, self powered
ehci1: <NEC uPD 720100 USB 2.0 controller> mem 0xfdafd000-0xfdafd0ff irq 18 at device 3.2 on pci4
ehci1: [GIANT-LOCKED]
ehci1: [ITHREAD]
usb7: EHCI version 1.0
usb7: companion controllers, 3 ports each: usb5 usb6
usb7: <NEC uPD 720100 USB 2.0 controller> on ehci1
usb7: USB revision 2.0
uhub7: <NEC EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb7
uhub7: 5 ports with 5 removable, self powered
umass4: <Western Digital External HDD, class 0/0, rev 2.00/1.02, addr 2> on uhub7
umass5: <ADS Technologies USB2.0 Drive Kit, class 0/0, rev 2.00/0.01, addr 3> on uhub7
umass5: Get Max Lun not supported (STALLED)
umass6: <ADS Technologies USB2.0 Drive Kit, class 0/0, rev 2.00/0.01, addr 4> on uhub7
umass6: Get Max Lun not supported (STALLED)
sym0: <875> port 0x9c00-0x9cff mem 0xfdafc000-0xfdafc0ff,0xfdafb000-0xfdafbfff irq 17 at device 4.0 on pci4
sym0: No NVRAM, ID 7, Fast-20, SE, parity checking
sym0: [ITHREAD]
pcib5: <ACPI Host-PCI bridge> on acpi0
pci128: <ACPI PCI bus> on pcib5
pci128: <multimedia, HDA> at device 1.0 (no driver attached)
acpi_tz0: <Thermal Zone> on acpi0
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FILTER]
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A
sio0: [FILTER]
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: [ITHREAD]
psm0: model IntelliMouse, device ID 3
cpu0: <ACPI CPU> on acpi0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
acpi_hpet0: <High Precision Event Timer> iomem 0xfe800000-0xfe8003ff on acpi0
device_attach: acpi_hpet0 attach returned 12
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc97ff,0xcc000-0xcdfff pnpid ORM0000 on isa0
ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppbus0: <Parallel port bus> on ppc0
ppbus0: [ITHREAD]
plip0: <PLIP network interface> on ppbus0
plip0: WARNING: using obsoleted IFF_NEEDSGIANT flag
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
ppc0: [GIANT-LOCKED]
ppc0: [ITHREAD]
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounter "TSC" frequency 2266665820 Hz quality 800
Timecounters tick every 1.000 msec
ad0: 78533MB <Hitachi HDS722580VLAT20 V32OA63A> at ata0-master UDMA100
acd0: DMA limited to UDMA33, device found non-ATA66 cable
acd0: DVDR <LITE-ON DVDRW SOHW-1633S/BS0S> at ata1-master UDMA33
acd1: CDRW <LITE-ON LTR-48327S/PTS1> at ata1-slave UDMA33
Waiting 5 seconds for SCSI devices to settle
GEOM_LABEL: Label for provider ad0s1a is ufsid/4a42cfbde524d087.
GEOM_LABEL: Label for provider ad0s1d is ufsid/4a42cfc21242e734.
GEOM_LABEL: Label for provider ad0s1e is ufsid/4a42cfbdfcdf27b1.
GEOM_LABEL: Label for provider ad0s1f is ufsid/4a42cfbd65525a3f.
GEOM_LABEL: Label for provider ad0s1g is ufsid/4a42cfbd75a68b18.
GEOM_LABEL: Label for provider ad0s2d is ufsid/4a42cfbdcada79a0.
GEOM_LABEL: Label for provider ad0s2e is ufsid/4a42cfc28b730061.
GEOM_LABEL: Label for provider ad0s2f is ufsid/4a42cfc236be6f59.
GEOM_JOURNAL: Journal 503343105: ad0s2g contains data.
GEOM_JOURNAL: Journal 503343105: ad0s2g contains journal.
GEOM_JOURNAL: Journal ad0s2g clean.
GEOM_LABEL: Label for provider ad0s2g.journal is ufsid/4a430e552079b936.
(probe12:sym0:0:5:0): TEST UNIT READY. CDB: 0 0 0 0 0 0
(probe12:sym0:0:5:0): CAM Status: SCSI Status Error
(probe12:sym0:0:5:0): SCSI Status: Check Condition
(probe12:sym0:0:5:0): UNIT ATTENTION asc:29,2
(probe12:sym0:0:5:0): SCSI bus reset occurred
(probe12:sym0:0:5:0): Retrying Command (per Sense Data)
sa0 at sym0 bus 0 target 5 lun 0
sa0: <HP C5713A H307> Removable Sequential Access SCSI-3 device
sa0: 40.000MB/s transfers (20.000MHz, offset 16, 16bit)
ch0 at sym0 bus 0 target 5 lun 1
ch0: <HP C5713A H307> Removable Changer SCSI-3 device
ch0: 40.000MB/s transfers (20.000MHz, offset 16, 16bit)
ch0: 6 slots, 1 drive, 0 pickers, 0 portals
da0 at umass-sim0 bus 0 target 0 lun 0
da0: <Initio HDT721010SLA360 1.43> Fixed Direct Access SCSI-0 device
da0: 40.000MB/s transfers
da0: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C)
da1 at umass-sim1 bus 1 target 0 lun 0
da1: <Initio WD1002FBYS-01A6B 1.43> Fixed Direct Access SCSI-0 device
da1: 40.000MB/s transfers
da1: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C)
da2 at umass-sim2 bus 2 target 0 lun 0
da2: <Initio WD1002FBYS-01A6B 1.43> Fixed Direct Access SCSI-0 device
da2: 40.000MB/s transfers
da2: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C)
da3 at umass-sim3 bus 3 target 0 lun 0
da3: <Initio HDT721010SLA360 1.43> Fixed Direct Access SCSI-0 device
da3: 40.000MB/s transfers
da3: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C)
da4 at umass-sim4 bus 4 target 0 lun 0
da4: <WD 1600BEVExternal 1.02> Fixed Direct Access SCSI-0 device
da4: 40.000MB/s transfers
da4: 152627MB (312581808 512 byte sectors: 255H 63S/T 19457C)
da5 at umass-sim5 bus 5 target 0 lun 0
da5: <WDC WD25 00JB-00GVC0 \\0000\\0000> Fixed Direct Access SCSI-0 device
da5: 40.000MB/s transfers
da5: 238475MB (488397168 512 byte sectors: 255H 63S/T 30401C)
da6 at umass-sim6 bus 6 target 0 lun 0
da6: <WDC WD25 00JB-00GVA0 \\0000\\0000> Fixed Direct Access SCSI-0 device
da6: 40.000MB/s transfers
da6: 238475MB (488397168 512 byte sectors: 255H 63S/T 30401C)
GEOM_MIRROR: Device mirror/gm-bacula launched (2/2).
GEOM_MIRROR: Device mirror/gm-media launched (2/2).
GEOM_MIRROR: Device mirror/gm-san launched (1/2).
GEOM_MIRROR: Device gm-san: rebuilding provider da5.
GEOM_JOURNAL: Journal 3034050800: mirror/gm-bacula contains data.
GEOM_JOURNAL: Journal 3034050800: mirror/gm-bacula contains journal.
GEOM_JOURNAL: Journal mirror/gm-bacula clean.
GEOM_JOURNAL: Journal 262277634: mirror/gm-media contains data.
GEOM_JOURNAL: Journal 262277634: mirror/gm-media contains journal.
GEOM_JOURNAL: Journal mirror/gm-media clean.
GEOM_LABEL: Label for provider da4s1 is ufsid/49273a95d669d784.
GEOM_LABEL: Label for provider da4s1 is ufs/backup.
GEOM_JOURNAL: Journal 2270030291: mirror/gm-san contains data.
GEOM_JOURNAL: Journal 2270030291: mirror/gm-san contains journal.
GEOM_JOURNAL: Journal mirror/gm-san clean.
(da6:umass-sim6:6:0:0): SYNCHRONIZE CACHE(10). CDB: 35 0 0 0 0 0 0 0 0 0
(da6:umass-sim6:6:0:0): CAM Status: SCSI Status Error
(da6:umass-sim6:6:0:0): SCSI Status: Check Condition
(da6:umass-sim6:6:0:0): ILLEGAL REQUEST asc:20,0
(da6:umass-sim6:6:0:0): Invalid command operation code
(da6:umass-sim6:6:0:0): Unretryable error
GEOM_MIRROR: Request failed (error=5). da6[READ(offset=0, length=512)]
GEOM_JOURNAL: BIO_FLUSH not supported by mirror/gm-san.
GEOM_LABEL: Label for provider mirror/gm-bacula.journal is ufsid/4a509cddbd500a7e.
GEOM_LABEL: Label for provider mirror/gm-media.journal is ufsid/4a3f26878cf7f367.
GEOM_LABEL: Label for provider mirror/gm-san.journal is ufsid/4a40c57f604c2e44.
Trying to mount root from ufs:/dev/ad0s1a
GEOM_LABEL: Label ufsid/4a42cfbde524d087 removed.
GEOM_LABEL: Label for provider ad0s1a is ufsid/4a42cfbde524d087.
GEOM_LABEL: Label ufsid/49273a95d669d784 removed.
GEOM_LABEL: Label ufsid/4a42cfbdfcdf27b1 removed.
GEOM_LABEL: Label for provider da4s1c is ufsid/49273a95d669d784.
GEOM_LABEL: Label for provider ad0s1e is ufsid/4a42cfbdfcdf27b1.
GEOM_LABEL: Label ufsid/4a42cfbd65525a3f removed.
GEOM_LABEL: Label for provider ad0s1f is ufsid/4a42cfbd65525a3f.
GEOM_LABEL: Label ufsid/4a42cfbd75a68b18 removed.
GEOM_LABEL: Label for provider ad0s1g is ufsid/4a42cfbd75a68b18.
GEOM_LABEL: Label ufsid/4a42cfbdcada79a0 removed.
GEOM_LABEL: Label for provider ad0s2d is ufsid/4a42cfbdcada79a0.
GEOM_LABEL: Label ufsid/4a42cfc28b730061 removed.
GEOM_LABEL: Label for provider ad0s2e is ufsid/4a42cfc28b730061.
GEOM_LABEL: Label ufsid/4a42cfc21242e734 removed.
GEOM_LABEL: Label for provider ad0s1d is ufsid/4a42cfc21242e734.
GEOM_LABEL: Label ufsid/4a42cfc236be6f59 removed.
GEOM_LABEL: Label for provider ad0s2f is ufsid/4a42cfc236be6f59.
GEOM_LABEL: Label ufsid/4a42cfbde524d087 removed.
GEOM_LABEL: Label ufsid/4a42cfbdfcdf27b1 removed.
GEOM_LABEL: Label ufsid/4a42cfbd65525a3f removed.
GEOM_LABEL: Label ufsid/4a42cfbd75a68b18 removed.
GEOM_LABEL: Label ufsid/4a42cfbdcada79a0 removed.
GEOM_LABEL: Label ufsid/4a430e552079b936 removed.
GEOM_LABEL: Label ufsid/4a42cfc28b730061 removed.
GEOM_LABEL: Label ufsid/4a42cfc21242e734 removed.
GEOM_LABEL: Label ufsid/4a42cfc236be6f59 removed.
GEOM_LABEL: Label ufsid/4a3f26878cf7f367 removed.
GEOM_LABEL: Label ufsid/4a40c57f604c2e44 removed.
GEOM_LABEL: Label ufsid/49273a95d669d784 removed.
fuse4bsd: version 0.3.9-pre1, FUSE ABI 7.8
GEOM_LABEL: Label ufsid/4a509cddbd500a7e removed.

--- On Fri, 8/14/09, Jeff Richards <bsd24x7 at yahoo.com> wrote:

From: Jeff Richards <bsd24x7 at yahoo.com>
Subject: Multiple USB drives stability question
To: freebsd-stable at freebsd.org
Date: Friday, August 14, 2009, 8:19 PM

Is there a practical limit on the number of active USB drives with FreeBSD?  I've had stability issues using multiple USB drives as storage.

My initial design goal was cheap, hot-swappable storage.  I am only using a 100MB network currently so throughput on the storage is not a problem as I can't push the data to/from the drives faster than what my network requests are.  

I first tried my setup on 7.0, then migrated to a newer PC, then upgraded to 7.2. 

I have the following USB drive setup:

1 320GB gmirror (320x2) + gjournal + ufs2
1 1TB gmirror (1TBx2) + gjournal + ufs2
1 150GB gjournal  + ufs2

I also have another 1TB gmirror (1TBx2) + gjournal but removed it.  The system crashed when I used these drives (bacula or bonnie++) so I pulled them to test on another system.

Recently my stability issue has been when I have been writing data to the 150GB gjournal drive from the 320GB gmirror device (USB device -> USB device).  It will be working fine, then all I/O stops on the 150GB drive.  The system remains responding to other USB devices etc. for a while.  I try rebooting and the system crashes with gjournal errors (didn't write down, but I will later).  

Every time this happens the 1TB gmirror comes up fine but one of the 320GB providers is missing.  No problem after 'gmirror forget' and 'gmirror insert'.  Everything rebuilds fine.  The 150GB gjournal drive is fine after a 'fsck -y'.

I do pair the gmirror drives to the same USB adapter.  Found out after initial testing with multiple USB adapters that they do not appear standard enough to cross adapters like I would for a production server at work to prevent SPOF with an adapter.

I have tried Linux as well with softraid and LVM2 on the same hardware.  It worked fine until I applied software updates and the udev took 30+ minutes to boot.  I went back to FreeBSD.  Even when I crashed I was back up in 2-5 minutes.

I can and will provide more detail if requested.  My concern is that the issue seems to continue no matter what hardware/OS changes I try.

Thanks in advance.

_______________________________________________
freebsd-stable at freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"

_______________________________________________
freebsd-stable at freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"