Namecache lock contention?

[Gleb Kurtsou]

On (28/01/2011 22:59), Ivan Voras wrote:
> On 28 January 2011 22:18, Gleb Kurtsou <gleb.kurtsou at gmail.com> wrote:
> 
> > You could try replacing rwlock with plain mutex to check if there are
> > priority propagation issues among readers/writers.
> 
> How would that manifest? (i.e. how would it be detectable)
Benchmark. If there are prio propagation issues mutex version should be
faster than original locking. Let me know if you need help with patch.

> > SX locks should also
> > work but would likely to be a considerable performance regression.
> 
> With mutexes I'd lose all shared (read) acquisitions so I doubt sx
> locks would do much more harm :)
Yeah, but sxlocks are more heavy weight. The question is what actual
readers/writers ratio in your test is, e.g. all namecache entries for a
dir may be purged on rename.

> > Finding out home much activity is there outside of storage/a/b/file
> > (sessions storage) could also be helpful.
> 
> Here's more information:
> 
> * The session storage (i.e. mostly file creates / writes in this
> particular workload) is on a separate file system than the core of the
> application (which only does reads)
Changing namecache lock to become per-file system would hardly improve
situation in general.

> * The dtrace output I've send is from around thirty seconds of
> operation, so around 2000 PHP runs. (PHP in this case is FastCGI, so
> the processes are persistent instead of constantly respawning). In
> these 2000 runs there have been around 20,000 rw-block events in
> cache_lookup - which is strange.
Are there rename, rmdir calls? - these purge namecache.
If cache is empty, VOP_LOOKUP acquires write lock to populate the cache.

> * Here's another dtrace output without rwlock mutex inlining, showing
> a different picture than what I've earlier thought: most rw-blocks
> events are in wlock! http://ivoras.net/stuff/rw-block-noinline.txt  --
> there are also some blocks without a rwlock function in the trace; I
> don't understand how rwlock inlining is implemented, maybe the readers
> are always inlined?
Add options RWLOCK_NOINLINE, recompiling with -O0 might also be good
idea.

> 
> Next step - find out how to make dtrace print files for which this happens.