cpuset and affinity implementation

[Jeff Roberson]

On Mon, 25 Feb 2008, Daniel Eischen wrote:

> On Mon, 25 Feb 2008, Jeff Roberson wrote:
>
>> On Mon, 25 Feb 2008, Daniel Eischen wrote:
>> 
>>> On Mon, 25 Feb 2008, Jeff Roberson wrote:
>>> 
>>>> On Mon, 25 Feb 2008, Alfred Perlstein wrote:
>>>> 
>>>>> Jeff, this is very cool.  I do have one issue though:
>>>>> 
>>>>> + * A thread may not be assigned to a a group seperate from other 
>>>>> threads in
>>>>> + * the process.  This is to remove ambiguity when the setid is queried 
>>>>> with
>>>>> + * a pid argument.  There is no other technical limitation.
>>>>> 
>>>>> Am I understanding things correctly such that within a process there
>>>>> can only be one "set"?
>>>>> 
>>>>> If so this restricts some of the benifits you get with sets and
>>>>> binding.
>>>>> 
>>>>> An example would be some sort of system with multiple CPUs where some
>>>>> are assigned specifically for pseudo-realtime processing and others are 
>>>>> for
>>>>> general control things such as cli, stats, etc.
>>>>> 
>>>>> In our case we would like to be able to run some threads on specific
>>>>> cpu sets, and other threads to be run anywhere on the control CPUs.
>>>>> 
>>>>> Can this be done with this API?
>>>> 
>>>> Individual threads can be bound to any cpu or group of cpus within the 
>>>> set. So if you just make a set that includes all cpus in the system you 
>>>> can then bind your realtime threads to specific cpus and the other 
>>>> threads to the remainder.  You will have to specifically bind each thread 
>>>> however.
>>>> 
>>>> The reason individual threads can't be assigned to groups is because 
>>>> cpuset_getid() for a pid wouldn't make sense then and I expect 
>>>> administrators to be mostly interested in managing groups of processes.
>>> 
>>> If the administrator sets up a set of CPUs specifically for
>>> real-time and another set for non-real-time, you may want to
>>> bind some threads to the real-time set, and leave other threads
>>> unbound (or even bound to the non-real-time set).  In this
>>> case, I think cpuset_getid() should either return the default
>>> cpuset of all cpus in the system, or the last cpuset to
>>> which the process was bound.
>>> 
>>> But regardless, I think binding a thread to a different
>>> processor set should be allowed and should override its
>>> inherent binding of the process' processor set.
>>> Hmm, I guess in this case, a subsequent binding of the
>>> process to a processor set should probably override any
>>> per-thread bindings.
>> 
>> I think we're getting into complex corner cases here which will only 
>> confuse the api and administrators.  I don't expect administrators will 
>> want to set groups to individual threads.  How would he even identify the 
>> individual thread?  And if he did, he could just as easily set masks on 
>> that thread along with others in the process.
>> 
>> I'm already a little nervous about how complicated this will be for 
>> programmers.   If we allowed each thread in a pid to be in its own set, 
>> we'd have to make cpuset_getid() return an array of ids.  I definitely 
>> don't want to do that.
>
> Solaris does seem to allow this BTW.

Solaris also doesn't allow a processor to be in more than one set.  It 
doesn't allow a thread to bind to a processor that's in a processor 
set.  It also doesn't seem provide a mechanism to query the set that a 
thread is in, so there is no ambiguity for the querying.  However, when 
you modify you have the option of retrieving the old set.  They must 
simply return the first one discovered.  We could do that but it doesn't 
seem very attractive.

Would people be in favor of binding threads to sets if it meant getting 
the id from a pid was not always 100% accurate?  Even though a thread may 
already restrict its set?

>From the pset_assign man page:

"Processors with LWPs bound to them using processor_bind(2) cannot be 
assigned to a new processor set. If this is attempted, pset_assign() will 
fail and set errno to EBUSY."

My cpuset design seems to be a lot more flexible.

Cheers,
Jeff

>
> -- 
> DE
>