Fine grain select locking.

[Attilio Rao]

Jeff Roberson wrote:
> 
> On Thu, 2 Aug 2007, Alfred Perlstein wrote:
> 
>> * Jeff Roberson <jroberson at chesapeake.net> [070802 17:52] wrote:
>>>
>>> I believe filedescriptor locking is the place where we are most lacking.
>>> The new sx helped tremendously.  However, this is still going to be a
>>> scalability limiter.  I have looked into both linux and solaris's 
>>> solution
>>> to this problem.  Briefly, linux uses RCU to protect the list, which is
>>> close to ideal as this is certainly a read heavy workload.  Solaris 
>>> on the
>>> other hand uses the actual file lock to protect the descriptor slot.  So
>>> they fetch the file pointer, lock it, and then check to see if they 
>>> lost a
>>> race with the slot being reassigned while they were acquiring the lock.
>>> This approach is perhaps better than rcu in many cases except when the
>>> descriptor set is expanded.  Then they have to lock every file in the 
>>> set.
>>
>> Certainly this is an extreme edge case... ?
> 
> Well that may be true, yes.  However, there are other problems with this 
> scheme.  For example, flags settings could be done entirely with cmpset, 
> without using a lock at all.  In most cases we're just setting a bit 
> which can be done with atomic_set.  When we're doing multiple operations 
> we could compute the value and attempt to est it in a loop.  So we can 
> totally eliminate locking the descriptor here.
> 
> We also could use atomic ops to protect the file descriptor reference 
> count.  This would eliminate another use of the FILE_LOCK().  I'm not 
> sure if it's possible to merge this with an approach that uses the 
> FILE_LOCK() to protect the descriptor table.  Although I've not thought 
> it all the way through.
> 
> If the ref count and flags were done with atomics the main consumer of 
> FILE_LOCK would actually be the unix domain socket garbage collection 
> code.  How's that for old unix baggage.  Do many programs actually pass 
> around descriptors these days?  inetd?  others?  It might be worth it to 
> lock this seperately from the file lock.

I'm sure I've alredy implemented it, but later I realized that there is 
a race with the p_fd field (if I got you right you are referring to the 
fdesc_mtx here), so we probabilly should better arrange those paths firstly.

Thanks,
Attilio