To SMP or not to SMP

[John Baldwin]

On Wednesday, January 09, 2013 10:00:31 am Barney Cordoba wrote:
> 
> --- On Wed, 1/9/13, sthaug at nethelp.no <sthaug at nethelp.no> wrote:
> 
> > From: sthaug at nethelp.no <sthaug at nethelp.no>
> > Subject: Re: To SMP or not to SMP
> > To: erichsfreebsdlist at alogt.com
> > Cc: barney_cordoba at yahoo.com, freebsd-net at freebsd.org, 
jack.vogel at gmail.com, atkin901 at gmail.com
> > Date: Wednesday, January 9, 2013, 9:32 AM
> > > > 4BSD runs pretty well with
> > an SMP kernel. I can test ULE and compare
> > > > easily. A no SMP kernel is problematic as the igb
> > driver doesn't seem
> > > > to work and my onboard NICs are, sadly, igb. 
> > > > 
> > > this is bad luck. I know of the kernels as I have had
> > SMP and single
> > > CPU machines since 4.x times.
> > 
> > I have had igb working with both SMP and non-SMP kernel for
> > at least a
> > year or two, 8.x-STABLE. No specific problems.
> > 
> > Steinar Haug, Nethelp consulting, sthaug at nethelp.no
> > 
> 
> Maybe a problem with "legacy interrupts" on more modern processors?
> I'm using an E5520 and while the NIC inits ok, it just doesnt seem to
> gen interrupts. I can't spend much time debugging it....
> 
> I notice that HAMMER kernels use MSI/X interrupts whether SMP is enabled
> or not, while i386 kernels seem to require APIC. Is there some physical
> reason for this?

MSI always requires APIC.  (MSI interrupts on x86 have to be delivered to a 
local APIC, no way to send them to 8259As.)  You can build an i386 kernel with 
device apic but without 'options SMP' which is akin to leaving SMP out of an 
amd64 kernel.

Removing SMP on x86 changes the following things:
- Spin mutexes just disable interrupts on the local CPU and don't use any
  atomic operations at all.  All other lock types work the same.
- atomic operations don't use the "lock" prefix so are cheaper.  However, the
  atomic op used for locks (cmpxchg) has an implicit "lock" prefix, so this
  isn't but so much of a gain.

-- 
John Baldwin