Fine-grained locking for POSIX local sockets (UNIX domain sockets)

[Kris Kennaway]

On Tue, May 09, 2006 at 11:13:02AM -0700, Greg 'groggy' Lehey wrote:
> On Monday,  8 May 2006 at 21:11:09 -0400, Kris Kennaway wrote:
> > On Mon, May 08, 2006 at 08:43:28PM -0400, Kris Kennaway wrote:
> >> On Mon, May 08, 2006 at 02:52:07AM -0400, Kris Kennaway wrote:
> >>> OK, David's patch fixes the umtx thundering herd (and seems to give a
> >>> 4-6% boost).  I also fixed a thundering herd in FILEDESC_UNLOCK (which
> >>> was also waking up 2-7 CPUs at once about 30% of the time) by doing
> >>> s/wakeup/wakeup_one/.  This did not seem to give a performance impact
> >>> on this test though.
> >>
> >> Turning down kern.hz from 1000 to 100 also made a big difference on 12
> >> CPUs (+6.1%).
> >>
> >> Note also that the system is no less than 40% idle during the runs (at
> >> any load), so the bottlenecks are serious.
> >
> > top -H shows the threads mostly in umtx state.
> 
> This doesn't lend much support to the idea that the gettimeofday()
> calls are a bottleneck.

There are at least several issues here:

* Factor of >two difference in performance across the board (all
loads) relative to Linux.  This may be general issues like
gettimeofday() on Linux vs FreeBSD; clearly there is something *very
big* to blame here.  Mysql does do *lots* of such calls, so the cost
of them is surely a component in performance, the only question is if
it's the main one.

* When you get some of the locking out of the way (per this thread)
FreeBSD has 44% better peak performance on Sven's test on amd64, but
tails off by about 33% at higher loads compared to unpatched.  I see
similar changes on 12-CPU E4500, but not as much performance gain (may
be due to other reasons).  i.e. optimizing the locking allows a new,
bigger bottleneck to suck on center stage.  This is the basis for my
observation about libthr at high loads.  It is not the same as the
above issue.

Kris
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-performance/attachments/20060509/e6bb07d8/attachment.pgp