Directory with large number of files

[Nathan Vidican]

In some passed projects of mine where large quantities of image files had to
be referenced I used a hierarchical tree structure based on the filename.
For example:

instead of '/disk/images/0018391.jpg' I used '/disk/images/10000/8/3/91.jpg'
instead of '/disk/images/abcdsomename.jpg' I used
'/disk/images/a/b/cdsomename.jpg'

In most cases it was to store large amounts of small image files to
reference from web-apps and such with a database key. Usually I tried to
work either numerically or alphabetically scaling to the database itself.
This also allows you to (if needed) use multiple disks/mount-points down the
line if the scale grows (ie: move /disk/images/a or /disk/images/10000 to a
dedicated disk/mount if room runs out or for performance reasons).

Not sure on the system-side what the advantages versus using a single large
directory will be - but I noted it was a much easier way to maintain things
and scaled very well (in one case had a directory with well over a million
images tied to a database of a product catalog, we broke it down by
department/section and then by numeric key and eventually had to dedicate
larger/faster storage to certain sections thus off-loading the work from
all-in-one and requiring minimal effort to do so).

Food for thought anyways - hope it helps.


-- 
Nathan Vidican
nathan at vidican.com


On Wed, Mar 10, 2010 at 11:51 AM, Janos Dohanics <web at 3dresearch.com> wrote:

> I have a directory with over 180,000 files in 4 subdirectories. Over
> 170,000
> of the files are in one of the subdirectories. The files are image files,
> mostly <50K. The machine has 2 GB RAM.
>
> What would be a reasonable setting for vfs.ufs.dirhash_maxmem? Is there a
> disadvantage to setting vfs.ufs.dirhash_maxmem too high?
>
> Is there any other setting to change to find files faster in a directory
> with
> such large number of files?
>
> Thank you for your advice.
>
> --
> Janos Dohanics
> _______________________________________________
> freebsd-questions at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "
> freebsd-questions-unsubscribe at freebsd.org"
>