Millions of small files: best filesystem / best options

[Peter Jeremy]

On 2012-May-28 15:21:57 +0200, Alessio Focardi <alessiof at gmail.com> wrote:
>I'm looking for some advice to efficiently pack millions of small files (200 bytes or less) over a freebsd fs.

"Millions of files" isn't an issue - any self-respecting filesystem
will manage that.  The "200 bytes or less" is a real problem.  The
best you will be able to do is a 4K/512 UFS1 filesystem (UFS1 because
the inodes are half the size) - that gives you a minimum of 640 bytes
(plus directory entry) per file (it would be 768 bytes for UFS2 and
1536 or 2048 bytes for ZFS).

I would suggest that no normal filesystem will be a good match for
your requirements.  IMHO, you would be better off storing your data as
an array of "file+metadata" objects within a single physical file,
together with some sort of index structure that suits your "file"
names.

-- 
Peter Jeremy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20120529/0d7ece4a/attachment.pgp