advice needed: zpool of 10 x (raidz2 on (4+2) x 2T HDD)

Tom Evans tevans.uk at googlemail.com
Wed Dec 2 13:13:16 UTC 2015


On Wed, Dec 2, 2015 at 11:34 AM, Zeus Panchenko <zeus at ibs.dn.ua> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> greetings,
>
> we deployed storage, and as it was filling until now, I see I need
> an advice regarding the configuration and optimization/s ...
>
> the main cause I decided to ask for an advice is this:
>
> once per month (or even more frequently, depends on the load I
> suggest) host hangs and only power reset helps, nothing helpful in log
> files though ... just the fact of restart logged and usual ctld activity
> <snip>
>   CPU: Intel(R) Xeon(R) CPU E5-2630L (2 package(s) x 6 core(s) x 2 SMT threads)
>   RAM: 128Gb
>  STOR: 3 x LSI SAS2308 PCI-Express Fusion-MPT SAS-2 (jbod)
>        60 x HDD 2T (ATA WDC WD20EFRX-68A 0A80, Fixed Direct Access SCSI-6 device 600.000MB/s)
>
> OS: FreeBSD 10.1-RELEASE #0 r274401 amd64
>
> to avoid OS memory shortage sysctl vfs.zfs.arc_max is set to 120275861504
>
> to clients, storage is provided via iSCSI by ctld (each target is file backed)
>
> zpool created of 10 x raidz2, each raidz2 consists of 6 geli devices and
> now looks so (yes, deduplication is on):
>
>> zpool list storage
> NAME            SIZE  ALLOC   FREE   FRAG  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
> storage         109T  33.5T  75.2T      -         -    30%  1.57x  ONLINE  -

You will need to examine zdb output to correctly determine the size of
your dedupe table. Assuming an average block size of 64kb, your DDT
will be approximately 167GB, ie well outside your RAM (my maths may be
off of course - ((33.5*(2**40)/(64*1024))*320)/(2**30)).

This article explains in detail:

http://constantin.glez.de/blog/2011/07/zfs-dedupe-or-not-dedupe

Cheers

Tom


More information about the freebsd-fs mailing list