Millions of small files: best filesystem / best options

[Bob Friesenhahn]

On Mon, 28 May 2012, Mark Felder wrote:

> ZFS is heavy, but if you have the resources it could possibly fit your needs 
> if tuned correctly. You can change the blocksize for any ZFS filesystem which 
> might help. It also deals with filesystems that have lots of files quite well 
> -- we have some customer backups that sprawl to 20 million+ files and ZFS 
> doesn't seem to care.

ZFS will work but its metadata size requirements will likely be twice 
the amount required by the actual file data.  It is not necessary to 
change the ZFS blocksize in order for it to work.

I have a directory in ZFS with 1 million files in one directory (not 
broken out into some heirarchy) that I use for testing application 
software.  ZFS does not mind a million files in one directory but 
applications can take quite a long time to obtain a file listing 
(fault of the application, not ZFS), especially if they want to sort 
that listing.

Bob
-- 
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/