Storing a lot of little files
cswiger at mac.com
Tue Jun 15 14:30:46 GMT 2004
> I need to store a lot (hundreds of millions) of very little files (from 8 bytes
> to 50K) in my filesystem. What's the best way to optimize it? Which newfs options can you
> recommend me?
Eric already provided an answer to this.
The type of problem you are considering is similiar to the way a Usenet news
server using tradspool works, or the way a large Squid proxy server would want
to layout it's cache. The techniques and advice used by INN and Squid for
configuring filesystems and laying out datafiles using multi-layer hierarchy
with perhaps a few hundred top-level directories, each containing a few
hundred second-level directories, etc are topics worth your consideration.
> Can UFS2 optiomize storage of little files? For example, put some
> files in one cluster - so don't let the 8 byte file to occupy a
> whole cluster of about 16K?
Good question. UFS breaks up 8K "clusters" into 1K fragments, so the amount
of wasted space due to instrinsic fragmentation is significantly less. That
being said, 8-byte files are too small to be kept individually with much
efficiency in any block-sized filesystem available to you, given the common
512-byte sector size of devices tends to constrain the minimum "cluster" size
one can set.
There are some useful ideas about how to deal with that from the IronDoc
project (Dave McCuster? [sp?] from Apple), but your requirements are difficult
to meet without taking some complex issues into consideration.
> Is it possible to allocate inodes dynamically (like in ReiserFS?),
> bacause I'm not sure I can predict accuratly number of little files,
> and inodes not to finish before disk get full.
No. Allocate many more inodes that you need if you can't guess accurately, or
else be prepared to backup, newfs using different parameters, and restore your
data when you run out.
> Does FreeBSD support file systems except of UFS?
FreeBSD supports FAT, ISO-9660, NTFS, ext2, probably others, although the
degree of support decreases in order of my listing.
> What about ReiserFS, XFS, JFS?
Nope. Partially licensing, partially lack of developer resources.
More information about the freebsd-current