SMP question w.r.t. reading kernel variables

[Clifton Royston]

On Tue, Apr 19, 2011 at 12:00:29PM +0000, freebsd-hackers-request at freebsd.org wrote:
> Subject: Re: SMP question w.r.t. reading kernel variables
> To: Rick Macklem <rmacklem at uoguelph.ca>
> Cc: freebsd-hackers at freebsd.org
> Message-ID: <201104181712.14457.jhb at freebsd.org>

[John Baldwin]
> On Monday, April 18, 2011 4:22:37 pm Rick Macklem wrote:
> > > On Sunday, April 17, 2011 3:49:48 pm Rick Macklem wrote:
...
> > All of this makes sense. What I was concerned about was memory cache
> > consistency and whet (if anything) has to be done to make sure a thread
> > doesn't see a stale cached value for the memory location.
> > 
> > Here's a generic example of what I was thinking of:
> > (assume x is a global int and y is a local int on the thread's stack)
> > - time proceeds down the screen
> > thread X on CPU 0                    thread Y on CPU 1
> > x = 0;
> >                                      x = 0; /* 0 for x's location in CPU 1's memory cache */
> > x = 1;
> >                                      y = x;
> > --> now, is "y" guaranteed to be 1 or can it get the stale cached 0 value?
> >     if not, what needs to be done to guarantee it?
> 
> Well, the bigger problem is getting the CPU and compiler to order the
> instructions such that they don't execute out of order, etc.  Because of that,
> even if your code has 'x = 0; x = 1;' as adjacent threads in thread X,
> the 'x = 1' may actually execute a good bit after the 'y = x' on CPU 1.

  Actually, as I recall the rules for C, it's worse than that.  For
this (admittedly simplified scenario), "x=0;" in thread X may never
execute unless it's declared volatile, as the compiler may optimize it
out and emit no code for it.


> Locks force that to sychronize as the CPUs coordinate around the lock cookie
> (e.g. the 'mtx_lock' member of 'struct mutex').
> 
> > Also, I see cases of:
> >      mtx_lock(&np);
> >      np->n_attrstamp = 0;
> >      mtx_unlock(&np);
> > in the regular NFS client. Why is the assignment mutex locked? (I had assumed
> > it was related to the above memory caching issue, but now I'm not so sure.)
> 
> In general I think writes to data that are protected by locks should always be
> protected by locks.  In some cases you may be able to read data using "weaker"
> locking (where "no locking" can be a form of weaker locking, but also a
> read/shared lock is weak, and if a variable is protected by multiple locks,
> then any singe lock is weak, but sufficient for reading while all of the
> associated locks must be held for writing) than writing, but writing generally
> requires "full" locking (write locks, etc.).

  What he said.  In addition to all that, lock operations generate
"atomic" barriers which a compiler or optimizer is prevented from
moving code across.

  -- Clifton

-- 
    Clifton Royston  --  cliftonr at iandicomputing.com / cliftonr at lava.net
       President  - I and I Computing * http://www.iandicomputing.com/
 Custom programming, network design, systems and network consulting services