SCHED_ULE should not be the default

[Steve Kargl]

On Thu, Dec 22, 2011 at 11:31:45AM +0100, Luigi Rizzo wrote:
> On Wed, Dec 21, 2011 at 04:52:50PM -0800, Steve Kargl wrote:
>> 
>> I have placed several files at
>> 
>> http://troutmask.apl.washington.edu/~kargl/freebsd
>> 
>> dmesg.txt      --> dmesg for ULE kernel
>> summary        --> A summary that includes top(1) output of all runs.
>> sysctl.ule.txt --> sysctl -a for the ULE kernel
>> ktr-ule-problem-kargl.out.gz 
>> 
>> 
>> Since time is executed on the master, only the 'real' time is of
>> interest (the summary file includes user and sys times).  This
>> command is run at 5 times for each N value and up to 10 time for
>> some N values with the ULE kernel.  The following table records
>> the average 'real' time and the number in (...) is the mean
>> absolute deviations. 
>> 
>> #  N         ULE             4BSD
>> # -------------------------------------
>> #  4    223.27 (0.502)   221.76 (0.551)
>> #  5    404.35 (73.82)   270.68 (0.866)
>> #  6    627.56 (173.0)   247.23 (1.442)
>> #  7    475.53 (84.07)   285.78 (1.421)
>> #  8    429.45 (134.9)   223.64 (1.316)
> 
> One explanation for taking 1.5-2x times is that with ULE the
> threads are not migrated properly, so you end up with idle cores
> and ready threads not running

That's what I guessed back in 2008 when I first reported the
behavior.  

http://freebsd.monkey.org/freebsd-current/200807/msg00278.html
http://freebsd.monkey.org/freebsd-current/200807/msg00280.html

The top(1) output at the above URL shows 10 completely independent
instances of the same numerically intensive application running
on a circa 2008 ULE kernel.  Look at the PRI column.  The high
PRI jobs are not only pinned to a cpu, but these are running at
100% WCPU.  The low PRI jobs seem to be pinned to a subset of the
available cpus and simply ping-pong in and out of the same cpus.
In this instance, there are 5 jobs competing for time on 3 cpus.

> Also, perhaps one could build a simple test process that replicates
> this workload (so one can run it as part of regression tests):
> 	1. define a CPU-intensive function f(n) which issues no
> 	   system calls, optionally touching
> 	   a lot of memory, where n  determines the number of iterations.
> 	2. by trial and error (or let the program find it),
> 	   pick a value N1 so that the minimum execution time
> 	   of f(N1) is in the 10..100ms range
> 	3. now run the function f() again from an outer loop so
> 	   that the total execution time is large (10..100s)
> 	   again with no intervening system calls.
> 	4. use an external shell script can rerun a process
> 	   when it terminates, and then run multiple instances
> 	   in parallel. Instead of the external script one could
> 	   fork new instances before terminating, but i am a bit
> 	   unclear how CPU inheritance works when a process forks.
> 	   Going through the shell possibly breaks the chain.

The tests at the above URL does essentially what you
propose except in 2008 the kzk90 programs were doing 
some IO.

-- 
Steve