cpuset and affinity implementation

[Jeff Roberson]

On Mon, 25 Feb 2008, Alfred Perlstein wrote:

> * Jeff Roberson <jroberson at chesapeake.net> [080224 02:39] wrote:
>> Please see:
>> http://people.freebsd.org/~jeff/cpuset.diff
>>
>> This is unfortunately intertwined with ULE's new CPU selection algorithm
>> so that code is in the patch as well.  Otherwise, this includes a simple,
>> ugly userland tool called cpuset and all of the kernel support required.
>> I have tested this by creating sets and subsets and modifying their cpu
>> masks under load.  I'm able to dynamically reprovision without issue.
>>
>> This doesn't have support for jails but the infrastructure is there.  It
>> also fails to modify sets if it would leave threads without a valid cpu
>> to run on.  I have not implemented a force option but it will be trivial
>> to do so.  The initial cpu set is also created before we know all_cpus so
>> it's faked up with all cpus set for now.
>>
>> I mostly want people to look at the interface in cpuset.h and make sure
>> they agree with it before I start polishing to commit.  I'm fairly happy
>> with the way the syscall api looks now.  The code itself ended up being
>> much more complicated than I'd hoped due to locking considerations.  Try
>> not to look at cpuset_setproc() ;).
>
>
> Jeff, this is very cool.  I do have one issue though:
>
> + * A thread may not be assigned to a a group seperate from other threads in
> + * the process.  This is to remove ambiguity when the setid is queried with
> + * a pid argument.  There is no other technical limitation.
>
> Am I understanding things correctly such that within a process there
> can only be one "set"?
>
> If so this restricts some of the benifits you get with sets and
> binding.
>
> An example would be some sort of system with multiple CPUs where some
> are assigned specifically for pseudo-realtime processing and others are for
> general control things such as cli, stats, etc.
>
> In our case we would like to be able to run some threads on specific
> cpu sets, and other threads to be run anywhere on the control CPUs.
>
> Can this be done with this API?

Individual threads can be bound to any cpu or group of cpus within the 
set.  So if you just make a set that includes all cpus in the system you 
can then bind your realtime threads to specific cpus and the other threads 
to the remainder.  You will have to specifically bind each thread however.

The reason individual threads can't be assigned to groups is because 
cpuset_getid() for a pid wouldn't make sense then and I expect 
administrators to be mostly interested in managing groups of processes.

It's really two different goals that are being served by this api.  You 
can think of the sets as more of an administrative tool.  While the 
private thread mask is a tool for the application programmer.  It just so 
happens that it was convenient under the hood to have it all managed in 
the same way.

Thanks,
Jeff

>
> -Alfred
>