4.8 ffs_dirpref problem

[Ken Marx]

Don Lewis wrote:
> On 31 Oct, To: kmarx at vicor.com wrote:
> 
>>On 31 Oct, Ken Marx wrote:
> 
> 
>>You may get better results if you change the minbfree calculation from
>>(avgbfree - avgbfree/4) to just (avgbfree).
>>
>>I'm somewhat tempted to change the calculation to:
>>	min(avgbfree, max(1, (avgbfree - avgbfree/4), (dirsize/fs->fs_bsize)))
>>where the last term works out to 4500 with your tunefs parameters.
> 
> 
> I tried a variation of this on my -CURRENT box and it benchmarked
> consistently worse.  I've got a "spare' 10 GB partition which first
> copied my /usr/ports/packages to, and then filled by repeatedly tarring
> my /usr/ports tree over to it.  The partition was 100% full, including
> the reserve space, after four iterations.
> 
> With minbfree set to max((avgbfree - avgbfree/4), 1) here are two
> iterations (the fifth line of timing data is for the 'rm -rf' command):
> 
>      1310.47 real         5.48 user       141.90 sys
>      1336.78 real         5.62 user       152.27 sys
>      1368.84 real         6.02 user       151.75 sys
>      1359.70 real         5.55 user       154.01 sys
>       423.44 real         2.25 user       107.26 sys
> 
>      1300.56 real         5.65 user       148.82 sys
>      1372.20 real         5.79 user       152.25 sys
>      1359.01 real         6.03 user       152.63 sys
>      1380.90 real         5.31 user       153.71 sys
>       437.22 real         2.20 user       105.61 sys
> 
> With minbfree set to
>  max(min(max(avgbfree - avgbfree / 4, dirsize / fs->fs_bsize),
>          avgbfree), 1)
> I get the following:
> 
>      1314.61 real         5.66 user       175.43 sys
>      1350.40 real         6.12 user       179.15 sys
>      1386.86 real         6.32 user       179.12 sys
>      1418.60 real         5.74 user       181.64 sys
>       508.67 real         2.67 user       119.66 sys
> 
>      1361.19 real         5.97 user       176.94 sys
>      1327.63 real         5.72 user       179.60 sys
>      1376.16 real         6.33 user       179.72 sys
>      1356.47 real         6.07 user       180.24 sys
>       462.67 real         2.30 user       119.18 sys
> 
> I'm using the newfs defaults, but dirsize is recalculated as the
> filesystem fills if the appropriate value is larger than what is
> calculated from the parameters set by newfs.
> 
> I suspect the problem is the large bimodal distribution in file size in
> my benchmark, with zillions of little files, but also a number of large
> package files and source distfiles.  The large files muck up the dirsize
> calculation because they are actually distributed across multiple
> cylinder groups and only the first maxbpg blocks are allocated in the
> original cylinder group.  This would be easy to account for in the
> avgfilesize * avgfpdir formula, but I don't know how to handle this in
> the curdirsize formula (other than the degenerate case where most files
> are larger than maxbpg).
> 
> Since I probably won't have time to get anything different tested before
> the -CURRENT code freeze, do you have any objections if I just MFC the
> code that I previously committed to -CURRENT?  It certainly seems to
> perform better than the original code which is still in 4-STABLE.
> 
> 

Don, any fine points you put on our corse level of testing here
are fine with us. I belive any of the versions you suggest will
keep us well out of the crippling behavior that originally brought
this up.

I was able to run a couple more tests here, and *belive* that the
fix to the hash table in vfs_bio.c will provide some relief
for cg block searches when things do fall into the linear search case.

k

-- 
Ken Marx, kmarx at vicor-nb.com
The entire team is behind the concept that we establish strategic alliances and 
identify trends in the context of the team building.
		- http://www.bigshed.com/cgi-bin/speak.cgi