UFS Subdirectory limit.

[Don Lewis]

On 27 Mar, Robert Watson wrote:

> FWIW, I regularly use directories with several hundred thousand files in
> them, and it works quite well for the set of operations I perform
> (typically, I only append new entries to the directory).  This is with a
> cyrus server hosting fairly large shared folders -- in Cyrus, a
> maildir-like format is used.  For example, the lists.linux.kernel
> directory references 430,000 individual files.  Between UFS_DIRHASH and
> Cyrus's use of a cache file, opening the folder primarily consists of
> mmap'ing the cache file and then doing lookups, which occur quite quickly.

I'm doing the same here and performance is OK for day to day operations,
but cloning the mail store is glacially slow.  I had to move my mail
store from a dieing disk to a replacement disk, and migrating about 5GB
of data took the better part of a day.  As the destination disk started
to fill up, the I/O rates dropped to very low levels and the CPU was
pegged at close to 100%, mostly system time.

The problem is that the allocation strategy of locating file inodes and
their data blocks in the same cylinder group as their parent directory
fails badly with directories of this size.  Once the cylinder group
fills, the search for free inodes and blocks gets very slow.

Using bigger cylinder groups would help a lot, especially considering
disk sizes these days, but both UFS and UFS2 have inconveniently small
maximum cylinder group sizes.  I think it would make sense to be able to
cluster groups of cylinder groups together for allocation purposes.

Something else that would seem to make a lot of sense would be to
implement something like the maxbpg limit for large directories that
would force the inode allocator to start allocating from another
cylinder group after a directory grows past a certain size.


> My workload doesn't currently require large numbers of directories
> referenced by a similar directory, but based on the results I've had with
> large numbers of files, I can say it likely would work fine subject to the
> ability for UFS to express it.