To SMP or not to SMP

[John Baldwin]

On Wednesday, January 09, 2013 06:42:15 PM Konstantin Belousov wrote:
> On Wed, Jan 09, 2013 at 03:54:43PM -0500, John Baldwin wrote:
> > On Wednesday, January 09, 2013 10:00:31 am Barney Cordoba wrote:
> > > --- On Wed, 1/9/13, sthaug at nethelp.no <sthaug at nethelp.no> wrote:
> > > > From: sthaug at nethelp.no <sthaug at nethelp.no>
> > > > Subject: Re: To SMP or not to SMP
> > > > To: erichsfreebsdlist at alogt.com
> > > > Cc: barney_cordoba at yahoo.com, freebsd-net at freebsd.org,
> > 
> > jack.vogel at gmail.com, atkin901 at gmail.com
> > 
> > > > Date: Wednesday, January 9, 2013, 9:32 AM
> > > > 
> > > > > > 4BSD runs pretty well with
> > > > 
> > > > an SMP kernel. I can test ULE and compare
> > > > 
> > > > > > easily. A no SMP kernel is problematic as the igb
> > > > 
> > > > driver doesn't seem
> > > > 
> > > > > > to work and my onboard NICs are, sadly, igb.
> > > > > 
> > > > > this is bad luck. I know of the kernels as I have had
> > > > 
> > > > SMP and single
> > > > 
> > > > > CPU machines since 4.x times.
> > > > 
> > > > I have had igb working with both SMP and non-SMP kernel for
> > > > at least a
> > > > year or two, 8.x-STABLE. No specific problems.
> > > > 
> > > > Steinar Haug, Nethelp consulting, sthaug at nethelp.no
> > > 
> > > Maybe a problem with "legacy interrupts" on more modern processors?
> > > I'm using an E5520 and while the NIC inits ok, it just doesnt seem to
> > > gen interrupts. I can't spend much time debugging it....
> > > 
> > > I notice that HAMMER kernels use MSI/X interrupts whether SMP is
> > > enabled or not, while i386 kernels seem to require APIC. Is there some
> > > physical reason for this?
> > 
> > MSI always requires APIC.  (MSI interrupts on x86 have to be delivered to
> > a local APIC, no way to send them to 8259As.)  You can build an i386
> > kernel with device apic but without 'options SMP' which is akin to
> > leaving SMP out of an amd64 kernel.
> > 
> > Removing SMP on x86 changes the following things:
> > - Spin mutexes just disable interrupts on the local CPU and don't use any
> >   atomic operations at all.  All other lock types work the same.
> > - atomic operations don't use the "lock" prefix so are cheaper.  However, the
> >   atomic op used for locks (cmpxchg) has an implicit "lock" prefix, so
> >   this isn't but so much of a gain.
> 
> I do not think that cmpxchg uses implicit lock, and Intel IA32 SDM supports
> this view. The absense of the lock prefix makes the instruction decode
> faster, and removes the locked bus cycle.
> 
> It is xchg which is implicitely locked, but we no longer use it for unlock.

Ah, that's true, for some reason I thought cmpxchg was like xchg in this
matter.

-- 
John Baldwin