you are in an fs with millions of small files
Robert Watson
rwatson at FreeBSD.org
Tue Jun 7 16:56:09 GMT 2005
On Tue, 7 Jun 2005, Eric Anderson wrote:
> Julian Elischer wrote:
>> what. all in one directory?
>>
>> I've only had up to 500,000 files in one directory on FreeBSD.
>
> The only problems I've had with a directory with millions of files is
> things like ls -al with attempt to sort the list, but the list doesn't
> fit into memory. Access to the files is of course very snappy.
Ditto. I regularly use directories with tens and hundreds of thousands of
entries as a result of manipulating very large folders with the Cyrus
server. I run into the following two classes of problems:
- Some appliations behave poorly with large trees. ls(1) is the classic
example -- sorting 150,000 strings is expensive, and should be avoided.
It also requires holding al the strings in memory rather than continuing
the iteration. fts ns bad about this, so many applications that use fts
suffer from this. With the sort issue, -f makes a big difference.
- Some operations become more expensive -- as directories grow, the cost
of adding new entries gets more expensive. You'll notice this fairly
substantailly if you untar a tar file with many entries in the same
directory -- early on, cost of insert for a new item is very cheap, but
it rapidly slows down from h thousands of inserts per second to hundreds
or less. I notice this if I restore a large Cyrus directory from
backup.
- UFS_DIRHASH really helps with large directory performance by reducing
the cost of lookup, but at the cost of memory. Make sure the box has
lots of memory.
All this said -- FreeBSD works really well for me with large file counts,
I rarely hit the edge cases where there is a problem. Most problems are
with applications, and when you are using more extreme file system
layouts, you typically are using applications customized for that andso
they do the right things.
Robert N M Watson
More information about the freebsd-current
mailing list