Storing a lot of little files
Julian Elischer
julian at elischer.org
Tue Jun 15 23:10:38 GMT 2004
On Tue, 15 Jun 2004, Cyrille Lefevre wrote:
> "Eugene" <el2000 at km.ru> wrote:
> > Hello freebsd-current,
> >
> > I need to store a lot (hundreds of millions) of very little files (from 8
> bytes
> > to 50K) in my filesystem.
>
> some times ago, there where something called "inode fs" (aka IFS).
> unfortunatelly, this was killed from -current (5.x) two years ago.
>
> more details here (google "freebsd +ifs +inode"):
>
> http://www.squid-cache.org/mail-archive/squid-dev/200101/0432.html
> http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/ufs/ifs/Attic/README
> http://lists.freebsd.org/pipermail/freebsd-fs/2003-June/000129.html
It's "supposed" to be coming back some time..
there are also "microfiles" (< 64 bytes)
which store the data in the inode.. You can simulate this
by using a symlink and using readlink(2) to instead of read(2)
when ever you encounter a symlink. (if you have source).
There were patches for microfiles floating around somewhere but
they were "pre-softupdates". I still think it would be a good idea.
/etc/malloc.conf uses this approach.
>
> > What's the best way to optimize it? Which newfs options can you
> > recommend me?
>
> so, the best way would be to have multi-level directories to reduce
> the number of entries in one directory whatever the underlying file
> system is (except, maybe, database-like filesystems).
>
> something like :
>
> /a/b/c/cfile
> /a/b/d/dfile
> /a/c/e/efile
> etc.
>
> using google "million +files +directory +fs" :
>
> http://aa11.cjb.net/sun_managers/2000/01/msg00303.html
>
> Cyrille Lefevre.
> --
> home: mailto:cyrille.lefevre at laposte.net
>
> _______________________________________________
> freebsd-current at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe at freebsd.org"
>
More information about the freebsd-current
mailing list