ZFS RaidZ2 with 24 drives?

[Wes Morgan]

On Fri, 25 Dec 2009, Solon Lutz wrote:

>> Depending on tuning, you can make it flush to disk more often.  It is also highly dependent on how much memory you have.
>
> At the moment: 4GB. I'm about to try upgrading it to 6GB.
>
> Why can't it work like this all the time:
>
> device     r/s   w/s    kr/s    kw/s wait svc_t  %b
> da0        0.0 1907.4     0.0 65494.8    0   0.6   6
> ad10     680.7   0.0 87132.0     0.0   35  43.7  92
>
>
> Effectively, it transfers 8-10MB/s! Took 24h for 1.2TB...
>
>> I know on my personal system, i see this happen a lot but it doesn't seem to have a hugely negative impact on
>> performance for what i use my machine for.  Depending on your setup, you may want to try various sysctl settings.  I
>> found that disabling prefetch can have a huge impact on some systems.
>
> Prefect is not enabled because of RAM < 4GB...

I have  my suspicions that this means your filesystem is heavily 
fragmented. I've had it happen to me on at least 3 pools, some of which 
were not even close to full, yet rebuilding the pool restored much of the 
performance. Hopefully with the block pointer rewrite support coming we 
will get some tools to address this. Right now I am not even aware of a 
tool that will check for fragmentation.