cpuset and affinity implementation

[Jeff Roberson]

On Mon, 25 Feb 2008, Brooks Davis wrote:

> On Sun, Feb 24, 2008 at 05:38:37PM -1000, Jeff Roberson wrote:
>> Hello,
>>
>> I have implemented a new api similar to processors sets on solaris.  This
>> allows you to assign processes to sets of cpus and dynamically change those
>> sets.  This is useful for provisioning purposes to add and remove cpu
>> resources for a particular process or group of processes.  This new
>> facility also supports binding secific threads to specific cpus which some
>> applications may want to do.  At some point in the future this will be
>> integrated with jail so you can restrict the cpus any jail is allowed to
>> use.
>>
>> This api should not be considered final and the 'cpuset' tool is quite
>> rough.  This also only works with ULE and is unfortunately intertwined with
>> a big ULE patch I've been working on.  The set management code is generic
>> but 4BSD doesn't contain the hooks to actually constrain threads.
>
> I took a look at the patch this morning.  The API looks like it's
> capable of doing what I need, at least at a first pass.  It looks like I
> should be able to implement the semantics currently employed by the Sun
> Grid Engine scheduler on Irix systems.
>
> The one thing I noticed that I found worrying was the recursion in
> cpuset_(test)update().  It wasn't immediately clear to me there there
> is anything to would prevent an arbitrarily deep hierarchy from being
> created and blowing the kernel stack.  I'm I missing something?

Yes, presently it can never be more than 3 levels deep.  Once we have 
jails the max would be 6 levels, unless you can make jails within jails.

There is presently now way for the user to create a cpuset that is a 
subset of another set.  So the three cpu sets are:

1)  Root set - immutable, all cpus, may be root of jail in which case root 
outside of the jail can change the set.
2)  cpuset - the set this process is a member of.
3)  mask - the anonymous set that is applied to an individual thread.

Did you look at the userland tool at all?  I think this needs the most 
improvement.  I basically just made something that would allow me to pass 
every possible parameter to the api.  Not exactly engineered for 
usability.

>
> -- Brooks
>