To SMP or not to SMP

[Konstantin Belousov]

On Mon, Jan 14, 2013 at 04:12:09PM -0600, Bryan Venteicher wrote:
> 
> 
> ----- Original Message -----
> > From: "John Baldwin" <jhb at freebsd.org>
> > To: freebsd-net at freebsd.org
> > Cc: "Konstantin Belousov" <kostikbel at gmail.com>, "Bryan Venteicher" <bryanv at daemoninthecloset.org>, "Peter Jeremy"
> > <peter at rulingia.com>
> > Sent: Monday, January 14, 2013 3:57:58 PM
> > Subject: Re: To SMP or not to SMP
> > 
> > On Monday, January 14, 2013 4:07:56 pm Konstantin Belousov wrote:
> > > On Mon, Jan 14, 2013 at 03:07:50PM -0500, John Baldwin wrote:
> > > > On Sunday, January 13, 2013 1:15:13 am Bryan Venteicher wrote:
> > > > > 
> > > > > ----- Original Message -----
> > > > > > From: "John Baldwin" <jhb at freebsd.org>
> > > > > > To: freebsd-net at freebsd.org
> > > > > > Cc: "Barney Cordoba" <barney_cordoba at yahoo.com>, "Peter
> > > > > > Jeremy"
> > <peter at rulingia.com>
> > > > > > Sent: Friday, January 11, 2013 9:39:17 AM
> > > > > > Subject: Re: To SMP or not to SMP
> > > > > > 
> > > > > > On Thursday, January 10, 2013 02:36:59 PM Peter Jeremy wrote:
> > > > > > > On 2013-Jan-07 18:25:58 -0800, Barney Cordoba
> > > > > > > <barney_cordoba at yahoo.com>
> > > > > > wrote:
> > > > > > > >I have a situation where I have to run 9.1 on an old
> > > > > > > >single core
> > > > > > > >box. Does anyone have a handle on whether it's better to
> > > > > > > >build a
> > > > > > > >non
> > > > > > > >SMP kernel or to just use a standard SMP build with just
> > > > > > > >the one
> > > > > > > >core?
> > > > > > > 
> > > > > > > Another input for this decision is kern/173322.  Currently
> > > > > > > on x86,
> > > > > > > atomic operations within kernel modules are implemented
> > > > > > > using calls
> > > > > > > to code in the kernel, which do or don't use lock prefixes
> > > > > > > depending
> > > > > > > on whethur the kernel was built as SMP.  My proposed change
> > > > > > > changes
> > > > > > > kernel modules to inline atomic operations but always
> > > > > > > include lock
> > > > > > > prefixes (effectively reverting r49999).  I'm appreciate
> > > > > > > anyone who
> > > > > > > feels like testing the impact of this change.
> > > > > > 
> > > > > > Presumably a locked atomic op is cheaper than a function call
> > > > > > then?
> > > > > >  The
> > > > > > current setup assumes the opposite.
> > > > > > 
> > > > > > I think we should actually do this for atomics in modules on
> > > > > > x86:
> > > > > > 
> > > > > > 1) If a module is built standalone, it should do whichever is
> > > > > > cheaper:
> > > > > >    a function call or always use "LOCK".
> > > > > > 
> > > > > > 2) If a module is built as part of the kernel build, it
> > > > > > should use
> > inlined
> > > > > >    atomics that match what the kernel does.  Thus, modules
> > > > > >    built with
> > a
> > > > > >    non-SMP kernel would use inlined atomic ops that do not
> > > > > >    use LOCK.
> > We
> > > > > >    have a way to detect this now (some HAVE_FOO #define added
> > > > > >    in the
> > past
> > > > > >    few years) that we didn't back when this bit of atomic.h
> > > > > >    was
> > > > > >    written.
> > > > > >
> > > > > 
> > > > > It would be nice to have the LOCK variants available even on UP
> > > > > kernels in non-hackish way. For VirtIO, we need to handle an
> > > > > guest
> > > > > UP kernel running on an SMP host. Whether this is an #define
> > > > > that
> > > > > forces the SMP atomics to be inlined, or if they're exposed
> > > > > with
> > > > > an _smp suffix.
> > > Could you please, clarify why does UP kernel needs it ?
> > > Shouldn't the hypervisor context switching provide neccessary
> > > serialization
> > > anyway ?
> > 
> > I thought this, too, but in the case of virtio you are presumably
> > sychronizing with other threads in the hypervisor itself which might
> > be running concurrently on another physical CPU.
> > 
> 
> Yes, that is the case to be concerned about. Although, thinking
> about this a bit more, in VirtIO (at least the current spec), all
> the shared fields are updated by either the host or guest, not
> both, so a UP kernel can get by without the LOCK, correct?
> 
I did not see the spec, so I cannot argue. On the other hands, the
barriers have nothing to do with shared access to the same memory
location. Barriers prevent seeing paradoxical results from memory
accesses, in particular, they ensure that compiler and hardware do
not reorder the memory access sequences in the unwanted way.

That said, I think that a model where some self-contained blob is
only writen by one agent, and only read by another, does not require
any barriers on x86 at all. The architecture specifies that the only
reordering the hardware is allowed to not hide are reads which could
pass writes.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 834 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-net/attachments/20130115/8f36c958/attachment.sig>