To SMP or not to SMP

[Bryan Venteicher]

----- Original Message -----
> From: "Konstantin Belousov" <kostikbel at gmail.com>
> To: "Bryan Venteicher" <bryanv at daemoninthecloset.org>
> Cc: "John Baldwin" <jhb at freebsd.org>, "Peter Jeremy" <peter at rulingia.com>, freebsd-net at freebsd.org
> Sent: Tuesday, January 15, 2013 4:42:16 AM
> Subject: Re: To SMP or not to SMP
> 
> On Mon, Jan 14, 2013 at 04:12:09PM -0600, Bryan Venteicher wrote:
> > 
> > 
> > ----- Original Message -----
> > > From: "John Baldwin" <jhb at freebsd.org>
> > > To: freebsd-net at freebsd.org
> > > Cc: "Konstantin Belousov" <kostikbel at gmail.com>, "Bryan
> > > Venteicher" <bryanv at daemoninthecloset.org>, "Peter Jeremy"
> > > <peter at rulingia.com>
> > > Sent: Monday, January 14, 2013 3:57:58 PM
> > > Subject: Re: To SMP or not to SMP
> > > 
> > > On Monday, January 14, 2013 4:07:56 pm Konstantin Belousov wrote:
> > > > On Mon, Jan 14, 2013 at 03:07:50PM -0500, John Baldwin wrote:
> > > > > On Sunday, January 13, 2013 1:15:13 am Bryan Venteicher
> > > > > wrote:
> > > > > > 
> > > > > > ----- Original Message -----
> > > > > > > From: "John Baldwin" <jhb at freebsd.org>
> > > > > > > To: freebsd-net at freebsd.org
> > > > > > > Cc: "Barney Cordoba" <barney_cordoba at yahoo.com>, "Peter
> > > > > > > Jeremy"
> > > <peter at rulingia.com>
> > > > > > > Sent: Friday, January 11, 2013 9:39:17 AM
> > > > > > > Subject: Re: To SMP or not to SMP
> > > > > > > 
> > > > > > > On Thursday, January 10, 2013 02:36:59 PM Peter Jeremy
> > > > > > > wrote:
> > > > > > > > On 2013-Jan-07 18:25:58 -0800, Barney Cordoba
> > > > > > > > <barney_cordoba at yahoo.com>
> > > > > > > wrote:
> > > > > > > > >I have a situation where I have to run 9.1 on an old
> > > > > > > > >single core
> > > > > > > > >box. Does anyone have a handle on whether it's better
> > > > > > > > >to
> > > > > > > > >build a
> > > > > > > > >non
> > > > > > > > >SMP kernel or to just use a standard SMP build with
> > > > > > > > >just
> > > > > > > > >the one
> > > > > > > > >core?
> > > > > > > > 
> > > > > > > > Another input for this decision is kern/173322.
> > > > > > > >  Currently
> > > > > > > > on x86,
> > > > > > > > atomic operations within kernel modules are implemented
> > > > > > > > using calls
> > > > > > > > to code in the kernel, which do or don't use lock
> > > > > > > > prefixes
> > > > > > > > depending
> > > > > > > > on whethur the kernel was built as SMP.  My proposed
> > > > > > > > change
> > > > > > > > changes
> > > > > > > > kernel modules to inline atomic operations but always
> > > > > > > > include lock
> > > > > > > > prefixes (effectively reverting r49999).  I'm
> > > > > > > > appreciate
> > > > > > > > anyone who
> > > > > > > > feels like testing the impact of this change.
> > > > > > > 
> > > > > > > Presumably a locked atomic op is cheaper than a function
> > > > > > > call
> > > > > > > then?
> > > > > > >  The
> > > > > > > current setup assumes the opposite.
> > > > > > > 
> > > > > > > I think we should actually do this for atomics in modules
> > > > > > > on
> > > > > > > x86:
> > > > > > > 
> > > > > > > 1) If a module is built standalone, it should do
> > > > > > > whichever is
> > > > > > > cheaper:
> > > > > > >    a function call or always use "LOCK".
> > > > > > > 
> > > > > > > 2) If a module is built as part of the kernel build, it
> > > > > > > should use
> > > inlined
> > > > > > >    atomics that match what the kernel does.  Thus,
> > > > > > >    modules
> > > > > > >    built with
> > > a
> > > > > > >    non-SMP kernel would use inlined atomic ops that do
> > > > > > >    not
> > > > > > >    use LOCK.
> > > We
> > > > > > >    have a way to detect this now (some HAVE_FOO #define
> > > > > > >    added
> > > > > > >    in the
> > > past
> > > > > > >    few years) that we didn't back when this bit of
> > > > > > >    atomic.h
> > > > > > >    was
> > > > > > >    written.
> > > > > > >
> > > > > > 
> > > > > > It would be nice to have the LOCK variants available even
> > > > > > on UP
> > > > > > kernels in non-hackish way. For VirtIO, we need to handle
> > > > > > an
> > > > > > guest
> > > > > > UP kernel running on an SMP host. Whether this is an
> > > > > > #define
> > > > > > that
> > > > > > forces the SMP atomics to be inlined, or if they're exposed
> > > > > > with
> > > > > > an _smp suffix.
> > > > Could you please, clarify why does UP kernel needs it ?
> > > > Shouldn't the hypervisor context switching provide neccessary
> > > > serialization
> > > > anyway ?
> > > 
> > > I thought this, too, but in the case of virtio you are presumably
> > > sychronizing with other threads in the hypervisor itself which
> > > might
> > > be running concurrently on another physical CPU.
> > > 
> > 
> > Yes, that is the case to be concerned about. Although, thinking
> > about this a bit more, in VirtIO (at least the current spec), all
> > the shared fields are updated by either the host or guest, not
> > both, so a UP kernel can get by without the LOCK, correct?
> > 
> I did not see the spec, so I cannot argue. On the other hands, the
> barriers have nothing to do with shared access to the same memory
> location. Barriers prevent seeing paradoxical results from memory
> accesses, in particular, they ensure that compiler and hardware do
> not reorder the memory access sequences in the unwanted way.
> 
> That said, I think that a model where some self-contained blob is
> only writen by one agent, and only read by another, does not require
> any barriers on x86 at all. The architecture specifies that the only
> reordering the hardware is allowed to not hide are reads which could
> pass writes.
>

I need to refresh from the IA32 manuals, but I'm pretty sure
LOCK doesn't end up matter for VirtIO since blobs are updated
by either the guest or host. But I think SMP still ends up being
required for x86 since atomic_load_acq_foo() on UP has only a
compiler membar.