Fine-grained locking for UNIX domain sockets: patch updated

[Robert Watson]

As part of Kris and Jeff's recent work on improving MySQL scalability on 
FreeBSD, I've updated my fine-grained locking patch for UNIX domain sockets to 
a more recent 7-CURRENT:

   http://www.watson.org/~robert/freebsd/netperf/20070224-uds-fine-grain.diff

This patch replaces the global mutex with a global rwlock, and introduces 
per-unpcb locks to protect the state of each socket at the UNIX domain socket 
layer.  Depending on the protocol operation, either the per-unpcb mutex is 
sufficient, or global read or write locks may also be required. The role of 
the global lock is to protect the topology of the connections between sockets, 
protect vnode v_socket pointers, and to prevent lock order issues in the event 
the per-unpcb lock of more than one socket must be acquired at a time.  For 
example, exclusive global locking is required during bind() and connect() to 
protect v_socket and change the topology of connected sockets, whereas recv() 
requires only the unpcb lock of the receiving socket.  send() may require 
either just an unpcb lock or a global lock if send() is implicitly connecting 
(sendto()) or disconnecting (PRUS_EOF) a socket.

This is basically the same patch that I have been maintaining since 2005 and 
posted at intermittent intervals when there have been reports of negative 
performance due to high contention on the UNIX domain socket subsystem lock. 
I continue to look for performance feedback, especially from workloads that 
are heavy consumers of UNIX domain sockets (such as MySQL).  My experience has 
been that it gives a small performance hit on workloads and in configurations 
without much contention/scalability, but a measurable performance benefit with 
anything more substantial.  Decrease in lower end workloads and configurations 
is at least in part due to the overhead of some additional locking vs. the 
benefits of reduced contention.  Something that I am particularly interested 
in looking for is little or no performance hit on UP.  This is a fairly 
heavily tested and refined patch, so the issue in deciding whether or not to 
commit this is its real-world effect on performance.

Robert N M Watson
Computer Laboratory
University of Cambridge