dirhash and dynamic memory allocation

[Miroslav Lachman]

Jeremy Chadwick wrote:
> On Fri, Oct 14, 2011 at 02:33:18PM +0200, Miroslav Lachman wrote:
>>
>>
>> Ivan Voras wrote:
>>> On 14/10/2011 11:20, Miroslav Lachman wrote:
>>>> Hi all,
>>>>
>>>> I tried some tuning of dirhash on our servers and after googlig a bit, I
>>>> found an old GSoC project wiki page about Dynamic Memory Allocation for
>>>> Dirhash: http://wiki.freebsd.org/DirhashDynamicMemory
>>>> Is there any reason not to use it / not commit it to HEAD?
>>>
>>> AFAIK it's sort-of already present. In 8-stable and recent kernels you
>>> can give huge amounts of memory to dirhash via vfs.ufs.dirhash_maxmem
>>> (but except in really large edge cases I don't think you *need* more
>>> than 32 MB), and the kernel will scale-down or free the memory if not
>>> needed.
>>
>> Is this change documented somewhere? Maybe it could be noticed on
>> DirhashDynamicMemory wiki page. Otherwise it seems as abandoned GSoC
>> project.
>
> There is no real form of "documentation" for this kind of change, but I
> do remember it being discussed on the mailing list at some point (an
> announcement or something?  I forget -- man it was a while ago).

I didn't mean real doc (man page or handbook), but just some official 
place (release notes?) stating the change of the dirhash behavior.

> There's probably a commit message.  I can dig it up if need be, but
> commit messages are often "Add Snake Wanglesnort support, I like
> samgyeopsal" and not much else.  I'm not even sure what man page would
> be relevant to document such pieces.  How about this?
>
> $ sysctl -a | grep dirhash
> vfs.ufs.dirhash_reclaimage: 5
> vfs.ufs.dirhash_lowmemcount: 0
> vfs.ufs.dirhash_docheck: 0
> vfs.ufs.dirhash_mem: 1075212
> vfs.ufs.dirhash_maxmem: 16777216
> vfs.ufs.dirhash_minsize: 2560
>
> $ for i in `sysctl -a | grep dirhash | cut -d: -f1`; do sysctl -d $i; done
> vfs.ufs.dirhash_reclaimage: max time in seconds of hash inactivity before deletion in low VM events
> vfs.ufs.dirhash_lowmemcount: number of times low memory hook called
> vfs.ufs.dirhash_docheck: enable extra sanity tests
> vfs.ufs.dirhash_mem: current dirhash memory usage
> vfs.ufs.dirhash_maxmem: maximum allowed dirhash memory usage
> vfs.ufs.dirhash_minsize: minimum directory size in bytes for which to use hashed lookup

Yes, I did this listing in the past when I tried tuning of dirhash on 
heavy loaded servers.

> No idea what docheck does.  But it looks like reclaimage (that's
> "reclaim age") is a pretty good indicator of what Ivan is describing.

I don't know the real purpose of docheck too, but it is mentioned in 10 
years old paper http://www.maths.tcd.ie/~dwmalone/p/bsdcon01.pdf "Make 
sure vfs.ufs.dirhash docheck set to 0." Nothing more.

> So basically, if you're worried about how much memory dirhash is taking
> up, you could log/graph vfs.ufs.dirhash_mem (unit = bytes).

I know this, my concern (and surprise) is that after some change, 
default value of dirhash_maxmem remains on 2MB, that's why I think that 
dynamic allocation GSoC was not commited.

>>> In effect, vfs.ufs.dirhash_maxmem is the upper limit - the kernel will
>>> use less and will free the allocated memory in low memory situations
>>> (which I've tried and it works).
>>>
>>>> And second question - is there any negative impact with higher
>>>> vfs.ufs.dirhash_maxmem? It stil defaults to 2MB (on FreeBSD 8.2) after
>>>
>>> Not that I know of.
>>>
>>>> 10 years, but I think we all are using bigger FS in these days with lot
>>>> of files and directories and 2MB is not enough.
>>>
>>> AFAIK I've changed it to autotune so it's configured to approximately 4
>>> MB on a 4 GB machine (and scales up) in 9.
>>
>> I don't have 9 installed to test it. Only 8-STABLE. Hope I will test
>> it soon.
>
> The above data I showed was taken from a RELENG_8 box.  The dirhash
> change in question I believe was either implemented in very early 8.x,
> or very late 7.x.  Again: I can dig up the commit if requested.

Data is same for one of our old 7.2 box. And not much different for 4.11 
(default values of min and max)

sysctl -a | grep dirhash
vfs.ufs.dirhash_minsize: 2560
vfs.ufs.dirhash_maxmem: 2097152
vfs.ufs.dirhash_mem: 1954584
vfs.ufs.dirhash_docheck: 0

And this is from already tuned 7.3

sysctl -a | grep dirhash
vfs.ufs.dirhash_reclaimage: 5
vfs.ufs.dirhash_lowmemcount: 20823
vfs.ufs.dirhash_docheck: 0
vfs.ufs.dirhash_mem: 5122176
vfs.ufs.dirhash_maxmem: 8388608
vfs.ufs.dirhash_minsize: 2560

When I read the wiki page about dynamic allocation, I thought there must 
be some bigger change in sysctl (no maxmem at all, or much higher 
default value). The current state requires me to manually raise maxmem 
to 8M (or more) on almost all our webservers and fileservers.

That's why I asked the question on the list, because nothing is 
indicating the change in dirhash. (Ivan already write about changes, so 
my question is answered :} )

Anyway - thank you for your reply.

Miroslav Lachman