svn commit: r241889 - in user/andre/tcp_workqueue/sys: arm/arm cddl/compat/opensolaris/kern cddl/contrib/opensolaris/uts/common/dtrace cddl/contrib/opensolaris/uts/common/fs/zfs ddb dev/acpica dev/...

Attilio Rao attilio at freebsd.org
Wed Oct 24 15:49:23 UTC 2012


On Wed, Oct 24, 2012 at 4:47 PM, Gleb Smirnoff <glebius at freebsd.org> wrote:
> On Wed, Oct 24, 2012 at 04:45:07PM +0100, Attilio Rao wrote:
> A> On Wed, Oct 24, 2012 at 4:43 PM, Gleb Smirnoff <glebius at freebsd.org> wrote:
> A> > On Wed, Oct 24, 2012 at 11:36:06AM -0400, John Baldwin wrote:
> A> > J> On Wednesday, October 24, 2012 11:24:22 am Attilio Rao wrote:
> A> > J> > On Wed, Oct 24, 2012 at 4:09 PM, Attilio Rao <attilio at freebsd.org> wrote:
> A> > J> > > On Wed, Oct 24, 2012 at 3:45 PM, John Baldwin <jhb at freebsd.org> wrote:
> A> > J> > >> On Wednesday, October 24, 2012 10:34:34 am Attilio Rao wrote:
> A> > J> > >>> On Wed, Oct 24, 2012 at 3:05 PM, John Baldwin <jhb at freebsd.org> wrote:
> A> > J> > >>> > On Tuesday, October 23, 2012 7:20:04 pm Andre Oppermann wrote:
> A> > J> > >>> >> On 24.10.2012 00:15, mdf at FreeBSD.org wrote:
> A> > J> > >>> >> > On Tue, Oct 23, 2012 at 7:41 AM, Andre Oppermann <andre at freebsd.org>
> A> > J> > >> wrote:
> A> > J> > >>> >> >> Struct mtx and MTX_SYSINIT always occur as pair next to each other.
> A> > J> > >>> >> >
> A> > J> > >>> >> > That doesn't matter.  Language basics like variable definitions should
> A> > J> > >>> >> > not be obscured by macros.  It either takes longer to figure out what
> A> > J> > >>> >> > a variable is (because one needs to look up the definition of the
> A> > J> > >>> >> > macro) or makes it almost impossible (because now e.g. cscope doesn't
> A> > J> > >>> >> > know this is a variable definition.
> A> > J> > >>> >>
> A> > J> > >>> >> Sigh, cscope doesn't expand macros?
> A> > J> > >>> >>
> A> > J> > >>> >> Is there a way to do the cache line alignment in a sane way without
> A> > J> > >>> >> littering __aligned(CACHE_LINE_SIZE) all over the place?
> A> > J> > >>> >
> A> > J> > >>> > I was hoping to do something with an anonymous union or some such like:
> A> > J> > >>> >
> A> > J> > >>> > union mtx_aligned {
> A> > J> > >>> >         struct mtx;
> A> > J> > >>> >         char[roundup2(sizeof(struct mtx), CACHE_LINE_SIZE)];
> A> > J> > >>> > }
> A> > J> > >>> >
> A> > J> > >>> > I don't know if there is a useful way to define an 'aligned mutex' type
> A> > J> > >>> > that will transparently map to a 'struct mtx', e.g.:
> A> > J> > >>> >
> A> > J> > >>> > typedef struct mtx __aligned(CACHE_LINE_SIZE) aligned_mtx_t;
> A> > J> > >>>
> A> > J> > >>> Unfortunately that doesn't work as I've verified with alc@ few months ago.
> A> > J> > >>> The __aligned() attribute only works with structures definition, not
> A> > J> > >>> objects declaration.
> A> > J> > >>
> A> > J> > >> Are you saying that the typedef doesn't (I expect it doesn't), or that this
> A> > J> > >> doesn't:
> A> > J> > >>
> A> > J> > >> struct mtx foo __aligned(CACHE_LINE_SIZE);
> A> > J> > >
> A> > J> > > I meant to say that such notation won't address the padding issue
> A> > J> > > which is as import as the alignment. Infact, for sensitive locks,
> A> > J> > > having just an aligned object is not really useful if the cacheline
> A> > J> > > gets shared.
> A> > J> > > In the end you will need to use explicit padding or use __aligned in
> A> > J> > > the struct definition, which cannot be used as a general pattern.
> A> > J> >
> A> > J> > The quickest way I see this can be made general is to have a specific
> A> > J> > struct defined in sys/_mutex.h like that
> A> > J> >
> A> > J> > struct mtx_unshare {
> A> > J> >        struct mtx lock;
> A> > J> >        char _pad[CACHE_LINE_SIZE - sizeof(struct mtx)];
> A> > J> > } __aligned(CACHE_LINE_SIZE);
> A> > J>
> A> > J> I think instead you want my union above that uses roundup2 in case a lock
> A> > J> eats up multiple cache lines:
> A> > J>
> A> > J> union mtx_foo {
> A> > J>      struct mtx lock;
> A> > J>      char junk[roundup2(sizeof(struct mtx), CACHE_LINE_SIZE)];
> A> > J> } __aligned_CACHE_LINE_SIZE;
> A> > J>
> A> > J> > then let mtx_* functions to accept void ptrs and cast them to struct
> A> > J> > mtx as long as the functions enter.
> A> > J>
> A> > J> Eh, that removes all compile time type checks.  That seems very dubious to me.
> A> >
> A> > I think that we should first get benchmarking results, and only then try
> A> > to evolve an API for cache aligned mutexes.
> A> >
> A> > As an option we can allocate mutexes from cache aligned uma zone dynamically,
> A> > to avoid all these syntax acrobatics.
> A>
> A> There are several objections to this. Quicker that came to my mind:
> A> - Some locks needs to be ready before UMA subsystem is setup
>
> I suppose there are not a lot of them, and those can be aligned by hand.

Actually I think they are quite a few. And however the whole point of
this is to avoid manual frobbing.

> A> - On arches where the KVA is already scarce (i386, powerpc, etc.) this
> A> is going to be completely overkill
>
> Do we really have that much mutexes on bss, that moving them to dynamic
> allocator would affect kva?

In my mind every lock that doesn't belong to a struct should be moved
to be mtx_unshared, really.

Attilio


-- 
Peace can only be achieved by understanding - A. Einstein


More information about the svn-src-user mailing list