CACHE_LINE_SIZE macro.

[Ian Lepore]

On Mon, 2012-11-05 at 10:11 -0700, Warner Losh wrote:
> On Nov 5, 2012, at 10:01 AM, Eitan Adler wrote:
> 
> > On 5 November 2012 11:49, Warner Losh <imp at bsdimp.com> wrote:
> >>> There has been some discussion recently about padding lock mutexs to
> >>> the cache line size in order to avoid false sharing of CPUs. Some have
> >>> claimed to see significant performance increases as a result.
> >> 
> >> Is that an out-of-kernel interface?
> >> 
> >> If we did that, we'd have to make it run-time settable, because there's no one right answer for arm and MIPS cpus: they are all different.
> > 
> > The discussion ended up with using a special parameter
> > CACHE_LINE_SIZE_LOCKS which is different than CACHE_LINE_SIZE. This is
> > necessary for other reasons as well (CACHE_LINE_SIZE_LOCKS may take
> > into account prefetching of cache lines, but CACHE_LINE_SIZE
> > wouldn't).
> > 
> > I think the "correct" thing to do here is choose a reasonable, but
> > not-always-correct CACHE_LINE_SIZE_LOCKS and make CACHE_LINE_SIZE a
> > per-board constant (or run time setting, or whatever works).  You
> > can't make it run-time settable as the padding is part of the ABI:
> > 
> > For more details see
> > http://comments.gmane.org/gmane.os.freebsd.devel.cvs/483696
> > which contains the original discussion.
> > 
> > Note - I was not involved.
> 
> this is a kernel-only interface, so compile time constants are fine there.  What user-land visible interfaces are affected by this setting?  The answer should be 'none'
> 
> Warner

When I commented on Attilio's recent checkins concerning padding of
locks to cache line size and the fact that the value changes per-cpu and
we're not well-positioned to handle that right now, his main concern was
modules matching the kernel.  I had suggested making the padding
conditional on SMP (because apparently there's no benefit to the padding
in a UP kernel), but then a module compiled for UP wouldn't work right
on an SMP kernel, and vice versa.  I'm not sure why that's a problem, my
solution to that would be "So then don't do that."

What scares me the most is the mushy definition of what CACHE_LINE_SIZE
really means.  There's nothing about the name that says "This may not be
the actual cache line size but it's probably close," but increasingly I
see people talking about it as if it had such a malleable meaning.  Is
that consistant with the existing uses in the code?  Is it a good idea?

-- Ian