Storing a lot of little files

[Brian Feldman]

On Tue, Jun 15, 2004 at 05:33:28PM +0400, Eugene wrote:
> Hello freebsd-current,
> 
>   I need to store a lot (hundreds of millions) of very little files (from 8 bytes
>   to 50K) in my filesystem.
>   What's the best way to optimize it? Which newfs options can you
>   recommend me?
>   Can UFS2 optiomize storage of little files? For example, put some
>   files in one cluster - so don't let the 8 byte file to occupy a
>   whole cluster of about 16K?
>   Is it possible to allocate inodes dynamically (like in ReiserFS?),
>   bacause I'm not sure I can predict accuratly number of little files,
>   and inodes not to finish before disk get full.

Are the files going to change a lot/at all?  The minimum allocation unit
in UFS is a fragment, so you are going to end up with quite a bit of waste
for the 8-byte files unless you choose something like 4KB blocks and
512 byte fragments.  Even then, you're wasting a lot of inode and
directory entry space when I believe you actually just want a flat space.
UFS2 allows for variable block and fragment sizes per inode, but I don't
know how finished the implmentation for that is (or how you would use
it).

Have you investigated using a database?  A database is better-suited for
this sort of task than a general-purpose filesystem.  If you were to,
for example, use Sleepycat DB (nee Berkeley DB) for this task, you could
create multiple tables in your database file each with different page
sizes to optimize the amount of space wasted (and pages needed) down
to the minimum (512 bytes is the smallest page size it supports).

>   Does FreeBSD support file systems except of UFS?
>   What about ReiserFS, XFS, JFS?

None of those are supported in any way that I'm aware of.  The FreeBSD
UFS with SoftUpdates should negate most needs for journalling, and the
actual physical storage and layout is already quite good.

-- 
Brian Fundakowski Feldman                           \'[ FreeBSD ]''''''''''\
  <> green at FreeBSD.org                               \  The Power to Serve! \
 Opinions expressed are my own.                       \,,,,,,,,,,,,,,,,,,,,,,\