file locking.

[Jeff Roberson]

I have been looking at file locking for 8.0 and have come up with a way to 
completely eliminate the file lock, reduce the size of struct file by 
~4 pointers (22%), remove the global list of files and associated lock, 
and restrict the scope of unp_gc while removing several race conditions.

The whole thing hinges on reducing the complexity and scope of unp_gc to 
remove several fields from struct file.  The remaining parts can be 
protected by atomics or are already protected by other locks.  f_count and 
f_type are now completely updated using atomics.  The ref counting with 
atomics results in significantly fewer atomics and cheaper fhold/fdrop. 
Protecting f_type was only complicated in cases where there were compound 
operations done on it, which are now implemented with atomic_cmpset_int 
loops.

The unix domain socket garbage collection was changed to scan the list of 
unp sockets rather than the list of all files.  This code is only 
responsible for finding dead cycles of unp sockets which reference each 
other.  Evaluating other descriptors is not necessary.  This allowed me to 
move f_gcflag and f_msgcount into unpcb.  This also removed a use of the 
global filelist allowing me to remove two pointers from struct file.

The only negative part of the new algorithm is that a back-pointer to 
the referencing struct file must be stored in any unix domain socket that 
is referenced via another in a rights message.  It is a slight layering 
violation but there is only ever 1 file for each unix domain socket so it 
is correct.  This is required because the garbage collection algorithm 
needs to know about the external references via the file.

The patch is available at:
http://people.freebsd.org/~jeff/fd.diff

pho and kris have both tested this and found it to be stable.  Kris has 
done some performance measurement and found it to be a win on 
microbenchmarks.  I can't imagine a case that would actually be slower, 
except perhaps endless loops of sysctl kern.file.  This also removes 
several sources of contention for multithreaded applications in 
particular, but also a global lock on allocating/freeing files.

This also resolves several cases where f_flag was not protected when it 
should be, as well as removing race conditions in the garbage collector 
code due to dropping locks, and fixing unprotected variables in the 
garbage collection code.

I intend to commit this soon after the 7.0 branch is made.  I also have 
the final revision of my improved select locking ready.

Thanks,
Jeff