Millions of small files: best filesystem / best options

[Doug Barton]

On 5/28/2012 10:01 AM, Alessio Focardi wrote:
> So in my case I would have to use -b 4096 -f 512
> 
> It's an improvement, but still is not ideal: still a big waste with 200 bytes files!

Are all of the files exactly 200 bytes? If so that's likely the best you
can do. The good news is that it's a big improvement (I've done similar
stuff in the past). You'll also want to tweak the -i (inode) value to
insure that you have sufficient inodes for the number of files you plan
to store. The default is not likely to be adequate for your needs.

Don't use journaling of any kind. If your primary access is going to be
reads, you should be Ok with softupdates. If you're going to be doing a
lot of writing consider no softupdates and async. Of course, all of this
assumes that you have a reasonable disk array setup (RAID or similar,
power backup, etc.) From the rest of your thread it seems that you have
this under control.

I think you already referenced this, but you also want to shard the file
system so that you don't have any one directory with more than 10k
files. Even with dirhash that's still a limitation. You don't have to
worry too much if you go a few files over 10k, but you don't want to
stretch it much more than that. Make sure you plan for growth over time.

hth,

Doug

-- 

    This .signature sanitized for your protection