did something change in SMP scheduling from 5.4 to 6.0?

[Lowell Gilbert]

Randy Schultz <schultz at sgi.com> writes:

> I am curious if something changed in the way extra cpus are utilized in 6.0
> vs. 5.4 in a SMP kernel.

Quite a lot, actually.

> My system is a dual-proc Xeon box.  When I was using 5.4 top would show
> both procs being used roughly the same, e.g. the "C" column usually had
> almost as many processes running on the second cpu as on the first.  On
> Jan 23 I did a cvsup and compiled a new 6.0-STABLE SMP kernel.  Now 
> watching top I rarely see the second proc being used, even though 
> roundrobin is enabled.  FWIW, here's some possibly relevant config bits:
> Dude ? sysctl -a|egrep smp
> kern.timecounter.smp_tsc: 0
> kern.smp.forward_roundrobin_enabled: 1
> kern.smp.forward_signal_enabled: 1
> kern.smp.cpus: 4
> kern.smp.disabled: 0
> kern.smp.active: 1
> kern.smp.maxcpus: 16
> Dude ? 
> Dude ? sysctl -a|egrep hyperth
> machdep.hyperthreading_allowed: 0
> 
> I've attached the output of a typical top with some builds running.  
> Unfortunately I don't have any saved data from 5.4.

What was the actual workload at that point?  It looks like there's
only one runnable process, and it's running, so there is no reason to
care which processor it's on.  

> I do see the second processor being used but those times are rare.  When it 
> does happen it seems almost exclusively when I'm seriously hammering the 
> system.  This makes me think something has changed with 6.0 behind-the-scenes
> such that if the system doesn't need the extra power then it sticks to cpu 0.  
> I poked around the release notes and handbook but failed to find anything
> on this topic.  Am I all wet and simply have something misconfigured or 
> indeed all is well with 6.x doing it's thing hugging cpu 0?

I'm not sure there's an issue here.  Spreading the work among two
processors might make slightly better use of cache space, but would
make power-saving modes less useful.  Now if you run two completely
independent CPU-intensive processes, you'll see both CPUs in use.