4.8 ffs_dirpref problem

Ken Marx kmarx at sploot.vicor-nb.com
Fri Oct 31 08:31:57 PST 2003

Kirk McKusick wrote:
> I know it takes a lot of time, but I would like to hear of the results
> when you do the initial loading of the filesystem using Don's code as
> that may well effect the set of choices that it has.

Ok. I've done this using 48k avgfilesize, and 1500 filesperdir.
I left our hashtable patch in.

I can give details, but Don's code seems to average a healthy
64-5sec/1.5gb untar. I.e., basically equivalent to the 4.4 code.  But
after 90% or so starts consuming more time - up to 90-130 seconds
(system time). This increase is not always monotonic. Timings in
the 60sec range to occur.  (I double-checked this, re-running
starting at 97%.)

The 4.4 dirpref code seems a bit better in this regime, staying
mostly in the 60-70 sec range.

I'm now starting from a newfs'd raid with the default settings,
and running Don's patch (with hashtable patch). Should be
done in 10hrs or so.

This matters to us, because we'd like to avoid having to newfs
all our production raids. 

On Thu, Oct 30, 2003 at 06:59:56PM -0800, Don Lewis wrote:
> On 30 Oct, Ken Marx wrote:
> > 
> > 
> > Don Lewis wrote:
> >> On 30 Oct, Ken Marx wrote:
> > Prehaps the dirpref patch lowers the frequency of having to search so much,
> > and hence exercises the hashtable less.
> I'm pretty sure that is the situation, but I wasn't sure if there was
> still enough hash table usage to make balancing it worthwhile.  It
> sounds like there isn't.

In this case at least. But is it still possible that some situation
might arise in which the quadratic search fails enough to warrant
having the more efficient hash for the multiple linear searches?

I guess I'm asking if folks are inclined to go for some hashtable
patch (in addition to Don's dirpref patch). 

I should probably compare hashtable vs. no-hashtable kernels,
with NO dirpref patch. I can do (not until tonight) if
this would help in deciding. Let me know.

> It looks like there is still about a 10% increase in system time for my
> modifications to dirpref versus reverting to the old algorithm.  It
> would be nice to know where the extra time was being spent, but that
> would require probably require some kernel profiling.

Right. In the high-capacity regime at least.

We can profile kernels here, but I'm running low on time I
can easily devote to this. Julian is due back in a few weeks
though. Let me know though, and I'll do whatever I can.

> It might also be interesting to play with the cylinder group selection
> criteria.  Should minbfree be 75% of avgbfree, 100% of avgbfree, or 125%
> of avgbfree?  Probably not anything over 100%, since the algorithm would
> fail if all the cylinder groups were evenly filled ...

(I'm way out of my depth here...)

Thanks again for the continued help with all this,

Ken Marx, kmarx at vicor-nb.com
Analyze progress on the family jewels!!
		- http://www.bigshed.com/cgi-bin/speak.cgi

More information about the freebsd-fs mailing list