Millions of small files: best filesystem / best options
Bob Friesenhahn
bfriesen at simple.dallas.tx.us
Mon May 28 17:14:51 UTC 2012
On Mon, 28 May 2012, Mark Felder wrote:
> ZFS is heavy, but if you have the resources it could possibly fit your needs
> if tuned correctly. You can change the blocksize for any ZFS filesystem which
> might help. It also deals with filesystems that have lots of files quite well
> -- we have some customer backups that sprawl to 20 million+ files and ZFS
> doesn't seem to care.
ZFS will work but its metadata size requirements will likely be twice
the amount required by the actual file data. It is not necessary to
change the ZFS blocksize in order for it to work.
I have a directory in ZFS with 1 million files in one directory (not
broken out into some heirarchy) that I use for testing application
software. ZFS does not mind a million files in one directory but
applications can take quite a long time to obtain a file listing
(fault of the application, not ZFS), especially if they want to sort
that listing.
Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
More information about the freebsd-fs
mailing list