Millions of small files: best filesystem / best options

[Bob Friesenhahn]

On Mon, 28 May 2012, Alessio Focardi wrote:

>> ZFS will work but its metadata size requirements will likely be twice
>> the amount required by the actual file data.  It is not necessary to
>> change the ZFS blocksize in order for it to work.
>
> So if I understand correctly you are suggesting that I use as few 
> directories as possible to save space (less metadata for them), but 
> still a single file will weight more in metadata than it's size?

No, I am not suggesting that at all.  It is just my own test case to 
see if my application can open a huge directory.

Each ZFS data block has associated redundant metadata data blocks and 
they can not consume less space than the logical sector size.

It is wise to subdivide the millions of files into a heirarchy so that 
ordinary application software can still work well with it (without 
being slow or crashing).

> That suggests that I cant have no metadatata suballocation, in any 
> case. Correct? So ... what's the size of metadata for a regular 
> file, even empty? A full block?

There is metadata for each data block and zfs always stores multiple 
copies of the metadata.  With 128K blocks and larger files, this 
overhead is no big deal but if your file is 200 bytes then at least 
three sectors are consumed just for the data itself and there is 
surely other metadata at the file level.  The storage space efficiency 
would not be very high.

The good news is that ZFS makes it very easy to test various scenarios 
to see how well solutions really work.

Bob
-- 
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/