Strange slowdown when cache devices enabled in ZFS

[Brendan Gregg]

On Wed, May 8, 2013 at 2:45 PM, Freddie Cash <fjwcash at gmail.com> wrote:

> On Wed, May 8, 2013 at 2:35 PM, Brendan Gregg <brendan.gregg at joyent.com>wrote:
>
>> Freddie Cash wrote (Mon Apr 29 16:01:55 UTC 2013):
>> |
>> | The following settings in /etc/sysctl.conf prevent the "stalls"
>> completely,
>> [...]
>> To feed at 160 Mbytes/sec, with an 8 Kbyte recsize, you'll need at least
>> 20,000 random read disk IOPS. How many spindles does that take? A lot. Do
>> you have a lot?
>>
>>
> 45x 2 TB SATA harddrives, configured in raidz2 vdevs of 6 disks each for a
> total of 7 vdevs (with a few spare disks).  With 2x SSD for log+OS and 2x
> SSD for cache.
>

What's the max random read rate? I'd expect (7 vdevs, modern disks) it to
be something like 1,000. What is your recsize? (or if it is tiny files,
then average size?).

On the other hand, if it's caching streaming workloads, then do those 2
SSDs outperform 45 spindles?

If you are getting 120 Mbytes/sec warmup, then I'm guessing it's either a
128 Kbyte recsize random reads, or sequential.

Brendan


> With plans to expand that out with another 45-disk JBOD next summer-ish
> (2014)
>
> With the settings above, I get 120 MBps of writes to the L2ARC until each
> SSD is over 90% full (boot), then it settles around 5-10 MBps while
> receiving snapshots from the other 3 servers.
>
> I guess I could change the settings to make the _boost 100-odd MBps and
> leave the _max at the default.  I'll play with the l2arc_write_* settings
> to see if that makes a difference with l2arc_norw enabled.
>
> --
> Freddie Cash
> fjwcash at gmail.com
>



-- 
Brendan Gregg, Joyent                      http://dtrace.org/blogs/brendan