Setting CPU affinity to process( Freebsd smp kernel)

[Terry Lambert]

From: Julian Elischer <julian at elischer.org>
>Kunze, Aaron wrote:
>> Thanks for the info.  The Linux equivalent would be sched_setaffinity
>> which takes a bitmask as input, allowing the user to define which
>> processors will run a particular thread.  Here's a link:
>> 
>> http://ibm5.ma.utexas.edu/cgi-bin/man-cgi?sched_setaffinity+2
>> 
>>>> There's a potential for 
>>>> conflict between the kernel's use of pinning and binding for kernel 
>>>> synchronization and the user space affinity model, which will be 
>> 
>> Can you elaborate on this?  Some of my colleagues and I are considering
>> tackling this and would like to avoid such pitfalls, if possible.
>
>[...]
>
>>>
>>> I know Solaris has processor_bind(2) and pset_bind(2):
>>>
>>>    http://docs.sun.com/app/docs/doc/816-5167/6mbb2jaeu?a=expand#P
>
>I prefer the linux model but it does limit you to some set number of procesors.
>it looks however that the solaris interface doesn't allow 'sets' of processors but
>just allows you to specify a single processor.

There's a lot bad to be said about the Linux interface, though it limits you to a
certain number of processors is not one of them.  It has the same limitations
on number of bits as FreeBSD's select(2) call, which is "none".

One big weakness of the Linux model is that it requires the application to pick
the affinity (to get use out of it, you effectively have to have a helper parent
that is knowledgable, and sets things up for a child process).

The good thing about the Solaris interface is that it's more or less starting to
become industry standard between vendors making machines with large
numbers of processors; HP also supports it.  It's likely to make standards track
at some point in the future, unless someone comes up with something better,
and gets one or more other people to implement it.

The documentation on the Solaris site is pretty bad, in that it doesn't tell you
where baby processor sets come from, initial state, or other issues that you'd
need to know in order to judge it fairly; the HP documentation is much better,
and the psrset(1M) manula page is most informative; here's a link:

    <http://docs.hp.com/en/B2355-60105/psrset.1M.html>

The most useful information is that you can iterate the processsors in a set,
there's always a processor set 0, and processor 0 always live in processor set
0, and at least one processor in processor set zero has to be willing to take I/O
interrupts and be available for scheduling LWPs (KSEs).  You can read the rest
at the link above for more details, but it should be clear that you can build any
set of CPUs and move them from set to set using this interface.

This is basically the old Mach "Processor Set" abstraction, revivified.

The major weakness of the Solaris/HP approach is that it does not permit any
migration of threads between processor sets (there is no hierarchy - Linux has
the same problem with its API).

If you consider a modern system with SMT ("hyprethreading"), multiple cores
on a die with shared L2, multiple dies in a package, and a shared L3 or memory
controller in the package, it becomes very obvious that any scalable approach
with a future needs to be able to have a hierarchy, let you control migration
between members in a set of sets, and let you control both negaffinity and
affinity in threads in a process (preferrably, both in the same process).

-- Terry