svn commit: r241889 - in user/andre/tcp_workqueue/sys: arm/arm cddl/compat/opensolaris/kern cddl/contrib/opensolaris/uts/common/dtrace cddl/contrib/opensolaris/uts/common/fs/zfs ddb dev/acpica dev/...

Thu Oct 25 13:51:03 UTC 2012

On Wednesday, October 24, 2012 11:14:18 pm Attilio Rao wrote:
> On Wed, Oct 24, 2012 at 9:13 PM, Attilio Rao <attilio at freebsd.org> wrote:
> > On Wed, Oct 24, 2012 at 7:14 PM, John Baldwin <jhb at freebsd.org> wrote:
> >> On Wednesday, October 24, 2012 11:41:24 am Attilio Rao wrote:
> >>> On Wed, Oct 24, 2012 at 4:36 PM, John Baldwin <jhb at freebsd.org> wrote:
> >>> > On Wednesday, October 24, 2012 11:24:22 am Attilio Rao wrote:
> >>> >> On Wed, Oct 24, 2012 at 4:09 PM, Attilio Rao <attilio at freebsd.org> wrote:
> >>> >> > On Wed, Oct 24, 2012 at 3:45 PM, John Baldwin <jhb at freebsd.org> wrote:
> >>> >> >> On Wednesday, October 24, 2012 10:34:34 am Attilio Rao wrote:
> >>> >> >>> On Wed, Oct 24, 2012 at 3:05 PM, John Baldwin <jhb at freebsd.org> wrote:
> >>> >> >>> > On Tuesday, October 23, 2012 7:20:04 pm Andre Oppermann wrote:
> >>> >> >>> >> On 24.10.2012 00:15, mdf at FreeBSD.org wrote:
> >>> >> >>> >> > On Tue, Oct 23, 2012 at 7:41 AM, Andre Oppermann <andre at freebsd.org>
> >>> >> >> wrote:
> >>> >> >>> >> >> Struct mtx and MTX_SYSINIT always occur as pair next to each other.
> >>> >> >>> >> >
> >>> >> >>> >> > That doesn't matter.  Language basics like variable definitions should
> >>> >> >>> >> > not be obscured by macros.  It either takes longer to figure out what
> >>> >> >>> >> > a variable is (because one needs to look up the definition of the
> >>> >> >>> >> > macro) or makes it almost impossible (because now e.g. cscope doesn't
> >>> >> >>> >> > know this is a variable definition.
> >>> >> >>> >>
> >>> >> >>> >> Sigh, cscope doesn't expand macros?
> >>> >> >>> >>
> >>> >> >>> >> Is there a way to do the cache line alignment in a sane way without
> >>> >> >>> >> littering __aligned(CACHE_LINE_SIZE) all over the place?
> >>> >> >>> >
> >>> >> >>> > I was hoping to do something with an anonymous union or some such like:
> >>> >> >>> >
> >>> >> >>> > union mtx_aligned {
> >>> >> >>> >         struct mtx;
> >>> >> >>> >         char[roundup2(sizeof(struct mtx), CACHE_LINE_SIZE)];
> >>> >> >>> > }
> >>> >> >>> >
> >>> >> >>> > I don't know if there is a useful way to define an 'aligned mutex' type
> >>> >> >>> > that will transparently map to a 'struct mtx', e.g.:
> >>> >> >>> >
> >>> >> >>> > typedef struct mtx __aligned(CACHE_LINE_SIZE) aligned_mtx_t;
> >>> >> >>>
> >>> >> >>> Unfortunately that doesn't work as I've verified with alc@ few months ago.
> >>> >> >>> The __aligned() attribute only works with structures definition, not
> >>> >> >>> objects declaration.
> >>> >> >>
> >>> >> >> Are you saying that the typedef doesn't (I expect it doesn't), or that this
> >>> >> >> doesn't:
> >>> >> >>
> >>> >> >> struct mtx foo __aligned(CACHE_LINE_SIZE);
> >>> >> >
> >>> >> > I meant to say that such notation won't address the padding issue
> >>> >> > which is as import as the alignment. Infact, for sensitive locks,
> >>> >> > having just an aligned object is not really useful if the cacheline
> >>> >> > gets shared.
> >>> >> > In the end you will need to use explicit padding or use __aligned in
> >>> >> > the struct definition, which cannot be used as a general pattern.
> >>> >>
> >>> >> The quickest way I see this can be made general is to have a specific
> >>> >> struct defined in sys/_mutex.h like that
> >>> >>
> >>> >> struct mtx_unshare {
> >>> >>        struct mtx lock;
> >>> >>        char _pad[CACHE_LINE_SIZE - sizeof(struct mtx)];
> >>> >> } __aligned(CACHE_LINE_SIZE);
> >>> >
> >>> > I think instead you want my union above that uses roundup2 in case a lock
> >>> > eats up multiple cache lines:
> >>>
> >>> Do you think locks can eat more than one cacheline? This would be
> >>> absolutely killer for performance.
> >>
> >> Not the lock cookie, but 'struct lock_object', etc. aren't entirely trivial.
> >> If you had a 32-bit platform with a 16-byte cache line size I wouldn't be
> >> surprised if the entire structure spilled over a cacheline.
> >
> > Cache line usually contains 8 words.
> > struct mtx is madeup only by 4 or 5 (depending if you are on 64 or 32 bits).
> > I think this is a no-concern and we should not encourage adding more
> > words to it anyway.

You cannot possibly assume that is true on every arm/mips/powerpc, etc.
processor in existence.  However, your approach doesn't require explicit
padding anyway, so it is a moot point.

> >>> > union mtx_foo {
> >>> >         struct mtx lock;
> >>> >         char junk[roundup2(sizeof(struct mtx), CACHE_LINE_SIZE)];
> >>> > } __aligned_CACHE_LINE_SIZE;
> >>> >
> >>> >> then let mtx_* functions to accept void ptrs and cast them to struct
> >>> >> mtx as long as the functions enter.
> >>> >
> >>> > Eh, that removes all compile time type checks.  That seems very dubious to me.
> >>>
> >>> Well right now fast path already has a fair amount of macros wrapping
> >>> the operations, which don't really enforce any type checks.
> >>
> >> Sure they do.  They still call a function that takes a 'struct mtx *' even
> >> if it isn't called in the fast path.  If you pass a 'struct sx *' to
> >> mtx_lock() it will fail to compile.  That needs to stay that way.
> >
> > I think that with some trickery using CTASSERT() and typeof() we may
> > be able to enforce sanity even with void * arguments.
> 
> I think I've had a better idea for this.
> In our locking scheme we already rely on the fact that lock_object
> will always be present in all our locking primitives and that it will
> be the first object of every locking primitives. This is an assumption
> we must live with in order to correctly implement lock classes. I
> think we can use the same concept in order to use the same KPI for the
> 2 different structures (struct mtx and struct mtx_unshare) and keep
> the compile time ability to find stupid bugs.
> 
> What I propose is that we assume mtx_lock remains always the second
> member of the struct mtx/mtx_unshare and no other lock is allowed to
> use such member name. This happens natually nowadays so there is no
> problem in having such a rule. What does that allows to do is to pass
> the address of the mtx_lock member to the underlying functions and
> from there we can get back the address of the mutex (because we assume
> that mtx_lock will be just after the first mandatory member
> lock_object).
> 
> Here is a patch that implements mtx_init() in the way I think about:
> http://people.freebsd.org/~attilio/mtx_unshare_poc.patch
> 
> this should give us all the desired effects. In this patch I've used
> volatile uintptr_t * but it can certainly be void * too, if you prefer
> less verbose.
> If you agree with this idea I can hack a patch right away.

You should just use member2struct() (or whatever that is called now).

Having the extra add/sub is annoying and the warning probably isn't quite as
clear, but probably doesn't matter but so much.

-- 
John Baldwin