svn commit: r242014 - head/sys/kern

[Jim Harris]

On Wed, Oct 24, 2012 at 12:16 PM, Andre Oppermann <andre at freebsd.org> wrote:

<snip>
>
>
> See also the discussion on svn-src-all regarding global struct mtx
> alignment.
>
> Thank you for proving my point. ;)
>
> Let's go back and see how we can do this the sanest way.  These are
> the options I see at the moment:
>
>  1. sprinkle __aligned(CACHE_LINE_SIZE) all over the place
>  2. use a macro like MTX_ALIGN that can be SMP/UP aware and in
>     the future possibly change to a different compiler dependent
>     align attribute
>  3. embed __aligned(CACHE_LINE_SIZE) into struct mtx itself so it
>     automatically gets aligned in all cases, even when dynamically
>     allocated.
>
> Personally I'm undecided between #2 and #3.  #1 is ugly.  In favor
> of #3 is that there possibly isn't any case where you'd actually
> want the mutex to share a cache line with anything else, even a data
> structure.

I've run my same tests with #3 as you describe, and I did see further
noticeable improvement.  I had a difficult time though quantifying the
effect it would have on all of the different architectures.  Putting
it in ULE's tdq gained 60-70% of the overall benefit, and was well
contained.

I agree that sprinkling all over the place isn't pretty.  But focused
investigations into specific locks (spin mutexes, default mutexes,
whatever) may find a few key additional ones that would benefit.  I
started down this path with the sleepq and turnstile locks, but none
of those specifically showed noticeable improvement (at least in the
tests I was running).  There's still some additional ones I want to
look at, but haven't had the time yet.

Thanks,

-Jim