ZFS RaidZ2 with 24 drives?

[Matt Simerson]

On Dec 15, 2009, at 7:34 AM, Solon Lutz wrote:

> Hi,
>
> are there any cons against building a RaidZ2 with 24 1.5TB drives?

Funny, I asked myself a similar question in July 2008. Except I had  
1.0TB drives.

$ ssh back01 zpool status
   pool: back01
  state: ONLINE
  scrub: none requested
config:

	NAME        STATE     READ WRITE CKSUM
	back01      ONLINE       0     0     0
	  da0       ONLINE       0     0     0
	  da1       ONLINE       0     0     0

errors: No known data errors


$ ssh back02 zpool status
   pool: back02
  state: ONLINE
status: One or more devices has experienced an error resulting in data
	corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
	entire pool from backup.
    see: http://www.sun.com/msg/ZFS-8000-8A
  scrub: none requested
config:

	NAME        STATE     READ WRITE CKSUM
	back02      ONLINE       0     0  934K
	  raidz1    ONLINE       0     0     0
	    da0     ONLINE       0     0     0
	    da1     ONLINE       0     0     0
	    da2     ONLINE       0     0     0
	    da3     ONLINE       0     0     0
	    da4     ONLINE       0     0     0
	    da5     ONLINE       0     0     0
	    da6     ONLINE       0     0     0
	  raidz1    ONLINE       0     0 1.83M
	    da8     ONLINE       0     0     0
	    da9     ONLINE       0     0     0
	    da10    ONLINE       0     0     0
	    da11    ONLINE       0     0     0
	    da12    ONLINE       0     0     0
	    da13    ONLINE       0     0     0
	    da14    ONLINE       0     0     0
	  raidz1    ONLINE       0     0 1.83M
	    da16    ONLINE       0     0     0
	    spare   ONLINE       0     0     0
	      da17  ONLINE       0     0     0
	      da7   ONLINE       0     0     0
	    da18    ONLINE       0     0     0
	    da19    ONLINE       0     0     0
	    da20    ONLINE       0     0     0
	    da21    ONLINE       0     0     0
	    da22    ONLINE       0     0     0
	spares
	  da15      AVAIL
	  da23      AVAIL
	  da7       INUSE     currently in use

errors: 241 data errors, use '-v' for a list

I tried several combinations and ran benchmarks against ZFS in various  
RAID-Z configs and finally determined that how I laid out the disks  
didn't affect performance much. That was well before ZFS v13 was  
committed, so there were many bug fixes and performance optimizations  
since then.

I deployed using the two configurations you see above. Both machines  
have a pair of Areca 1231ML RAID controllers with super-sized BBWC  
(battery backed write cache). On back01, each controller presents a 12- 
disk RAID-5 array and ZFS concatenates them into the zpool you see  
above. On back02, the RAID controller is configured in JBOD mode and  
disks are pooled as shown.

In 17 months of production, the ZFS pool on back02 has required  
maintenance several times, including being down for days while a scrub  
was being run. Yes, days. Several times. We've had a couple data  
center power outages, and the only ZFS equipped backup servers that  
had any issue was back02. The last scrub failed to fixed the data  
errors. IIRC, the RAID controller write cache is not active in JBOD  
mode. That could explain why back02 had problems and the rest of my  
ZFS servers did not.

When another disk in back02 fails, I'll move all the data off back02  
and rebuild the disk arrays using hardware RAID. In addition to having  
zero disk errors, zero hardware problems, and zero ZFS data errors,  
the ZFS backup servers deployed on top of hardware RAID are much  
faster. How much faster?

In the past 3 days, I have had a cleanup process running that prunes  
stale backups. On back01, the process has cleaned up 4TB of disk  
space. On back02, it has only cleaned up 1.2TB. These cleanup  
processes run while the machines are performing their 'normal' duties.  
On average, the back02 processes take about 3-4x longer to run on  
back02.  It's not for lack of resources either. These are dual quad- 
cores with 16GB of RAM each.

YMMV.

Matt


> In some old postings floating around the net a limit of 9 drives
> is recommended.
> Does this still apply to the current ZFS in 8.0?
>
>
> Best regards,
>
> Solon Lutz
>
>
> +-----------------------------------------------+
> | Pyro.Labs Berlin -  Creativity for tomorrow   |
> | Wasgenstrasse 75/13 - 14129 Berlin, Germany   |
> | www.pyro.de - phone + 49 - 30 - 48 48 58 58   |
> | info at pyro.de - fax + 49 - 30 - 80 94 03 52    |
> +-----------------------------------------------+
>
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"