rarely changing process-wide data vs threads

[Rui Paulo]

On Nov 23, 2014, at 15:14, Mateusz Guzik <mjguzik at gmail.com> wrote:
> 
> Currently we have some things frequently accessed which require
> locking, even though they very rarely change.
> 
> This includes:
> - cwd, root, jdir vnodes
> - resource limits
> 
> File lookup typically requires us to vref and unref cwd and root dir and
> locking filedesc lock shared which competes with fd open/close in other
> threads.
> 
> Any resource limit checks requires taking PROC_LOCK, which is an
> exclusive lock.
> 
> Turns out we already have a nice solution which only needs some minor
> refining and it was used to manage credentials:
> 
> Each thread has a reference on active credentials and has its own
> pointer. When credentials are updated, a new structure is allocated and
> threads check that they got the right pointer on syscall boundary. If
> they got the wrong one, they lock PROC_LOCK and update.
> 
> We can make this more general to suit other needs with an introduction
> of 'generation' counter and optionally an rwlock instead of using
> PROC_LOCK. If 'generation' is unequal to what is set in the process,
> at least one of creds/dirs/rlimits/$something needs updating and we can
> take the lock and iterate over structs.

Right, this is the same model used by the routing table.

> This may pose some concern since it may seem this introduces a window
> where given thread uses stale data while a concurrently executing thread
> uses new one.

Likewise there's a small race for the networking stack.

> This window is already present for all users that I can see.
> 
> During file lookups filedesc lock is only temporarily held (and current
> code even has a possible use after free since it does not start with
> refing root vnode while fdp is locked so it can be freed/recycled).
> 
> resource limits are inherently racy anyway. proc lock is held only for a
> short them to read them, that's it.
> 
> As such, I don't believe this approach introduces any new windows
> (although it extends already existing ones).

I agree.

> When it comes to implementation of this concept for dir vnodes, one
> would need to split current struct filedesc. chdir in threaded processes
> would be more expensive since new struct would have to be allocated and
> vnodes vrefed, but chdirs are way less frequent than lookups so it
> should be worth it anyway.

I agree.  A lookup is a different operation but most of the time a chdir is followed by a lookup, so if we optimise the lookup case the end result might still be better.

> There is also a note on filedescs shared between processes. In such
> cases we would abandon this optimisation (dir struct can have a flag to
> note cow is not suitable and lookups need to vref like they do now).

Are you talking about your optimisation or something that's already there?

--
Rui Paulo