ZFS obn FreeBSD hardware model for 48 or 96 sata3 paths...

[Freddie Cash]

On Mon, Sep 19, 2011 at 12:07 PM, Jason Usher <jusher71 at yahoo.com> wrote:

> --- On Sat, 9/17/11, Bob Friesenhahn <bfriesen at simple.dallas.tx.us> wrote:
>
> > 150KB is a relatively small file size given that the
> > default zfs blocksize is 128KB.  With so many files you
> > should definitely max out RAM first before using SSDs as a
> > l2arc.  It is important to recognize that the ARC cache
> > is not populated until data has been read.  The cache
> > does not help unless the data has been accessed several
> > times. You will want to make sure that all metada and
> > directories are cached in RAM.  Depending on how the
> > files are used/accessed you might even want to intentionally
> > disable caching of file data.
>
> How does one make sure that all metadata and directories are cached in RAM?
>  Just run a 'find' on the filesystem, or a 'du' during the least busy time
> of day ?  Or is there a more elegant, or more direct way to read all of that
> in ?
>

That should work to "prime" the caches.  Or you can just let the system
manage it automatically, adding data to the ARC/L2ARC as it's read/accessed.
 The end result of that would be much more in line with how the data is
actually used.


> Further, if this (small files, lots of them) dataset benefits a lot from
> having the metadata and dirs read in, how can I KEEP that data in the cache,
> but not cache the file data (as you suggest, above) ?


There are ZFS properties for this (primarycache aka ARC; secondarycache aka
L2ARC) which can be set on a per-filesystem basis (and inherited).  These
can be set to "all", "metadata", or "data".


> Can I explicitly cache metadata/dirs in RAM, and cache file data in L2ARC ?


No.  Data that does not go into the ARC can never go into the L2ARC.  IOW,
if you set primarycache=metadata and secondarycache=data, you will never see
anything in L2ARC.

At least, that's the understanding I've come to based on posts on the
zfs-discuss mailing list.  And it does jive with what I was seeing on our
storage servers.

It's too bad, because it would be a nice setup, ordered from fastert to
slowest:  ARC for metadata, L2ARC for file data, pool for permanent storage.

-- 
Freddie Cash
fjwcash at gmail.com