UFS Subdirectory limit.

[Scott Long]

David Schultz wrote:
> On Sat, Mar 26, 2005, David Malone wrote:
> 
>>There was a discussion on comp.unix.bsd.freebsd.misc about two weeks
>>ago, where someone had an application that used about 150K
>>subdirectories of a single directory. They wanted to move this
>>application to FreeBSD, but discovered that UFS is limited to 32K
>>subdirectories, because UFS's link count field is a signed 16 bit
>>quantity. Rewriting the application wasn't an option for them.
>>
>>I had a look at how hard it would be to fix this. The obvious route
>>of increasing the size of the link count field is trickly because
>>it means changing the struct stat, which has a 16 bit link count
>>field. This would imply ABI breakage, though it might be worth it.
> 
> 
> Why not just...
> 
> - make a new st_nlink field that's 32 bits and put it in the spare
>   32-bit field in struct stat
> 
> - rename the old st_nlink to st_onlink and leave it at 16 bits
> 
> - the kernel would fill in st_onlink with max(st_nlink,SHORT_MAX)

I thought that we already discussed this in the past year.  There are
significant compatibility concerns here.  What happens if you use an
old fsck binary on a new filesystem?  Since you haven't changed the
magic, it has no way of knowing that nlink needs to be handled
differently.  It would make it impossible to share a filesystem between
different versions of FreeBSD, let alone any other BSD.  It's hard to
justify bumping the magic for just a change like this, since it would
basically mean that you've created UFS3.  Also, the more important
concern is that large directories simply don't scale in UFS.  Lookups
are a linear operation, and while DIRHASH helps, it really doesn't scale
well to 150k entries.  I think the reason that there isn't more pressure
to fix the nlink size is because most people realize that it just won't
provide any real benefit.  It would be much more worthwhile to introduce
a UFS3 that uses a more efficient directory layout (B-tree?) to provide
real value to increasing the nlink limitation.

Scott