Fine grain select locking.

[Jeff Roberson]

On Thu, 2 Aug 2007, Alfred Perlstein wrote:

> * Jeff Roberson <jroberson at chesapeake.net> [070802 17:52] wrote:
>>
>> I believe filedescriptor locking is the place where we are most lacking.
>> The new sx helped tremendously.  However, this is still going to be a
>> scalability limiter.  I have looked into both linux and solaris's solution
>> to this problem.  Briefly, linux uses RCU to protect the list, which is
>> close to ideal as this is certainly a read heavy workload.  Solaris on the
>> other hand uses the actual file lock to protect the descriptor slot.  So
>> they fetch the file pointer, lock it, and then check to see if they lost a
>> race with the slot being reassigned while they were acquiring the lock.
>> This approach is perhaps better than rcu in many cases except when the
>> descriptor set is expanded.  Then they have to lock every file in the set.
>
> Certainly this is an extreme edge case... ?

Well that may be true, yes.  However, there are other problems with this 
scheme.  For example, flags settings could be done entirely with cmpset, 
without using a lock at all.  In most cases we're just setting a bit which 
can be done with atomic_set.  When we're doing multiple operations we 
could compute the value and attempt to est it in a loop.  So we can 
totally eliminate locking the descriptor here.

We also could use atomic ops to protect the file descriptor reference 
count.  This would eliminate another use of the FILE_LOCK().  I'm not sure 
if it's possible to merge this with an approach that uses the FILE_LOCK() 
to protect the descriptor table.  Although I've not thought it all the way 
through.

If the ref count and flags were done with atomics the main consumer of 
FILE_LOCK would actually be the unix domain socket garbage collection 
code.  How's that for old unix baggage.  Do many programs actually pass 
around descriptors these days?  inetd?  others?  It might be worth it to 
lock this seperately from the file lock.

Anyway, these things need to be explored for 8.0.  With more cores and 
more multi-threaded applications file desc locking is one of the first 
points of contention as evidenced by the lock profiles and the 
sophistication of the solutions in other kernels.

Jeff


>
> I could see it happening if we started with low limits, but perhaps
> by keeping counters/stats we could tell people how to tune their
> systems, or even autotune them.
>
> -Alfred
>