cpuset and affinity implementation

[Jeff Roberson]

On Tue, 26 Feb 2008, Nick Evans wrote:

> On Sun, 24 Feb 2008 00:40:31 -1000 (HST)
> Jeff Roberson <jroberson at chesapeake.net> wrote:
>
>> Please see:
>> http://people.freebsd.org/~jeff/cpuset.diff
>>
>> This is unfortunately intertwined with ULE's new CPU selection algorithm
>> so that code is in the patch as well.  Otherwise, this includes a simple,
>> ugly userland tool called cpuset and all of the kernel support required.
>> I have tested this by creating sets and subsets and modifying their cpu
>> masks under load.  I'm able to dynamically reprovision without issue.
>>
>> This doesn't have support for jails but the infrastructure is there.  It
>> also fails to modify sets if it would leave threads without a valid cpu
>> to run on.  I have not implemented a force option but it will be trivial
>> to do so.  The initial cpu set is also created before we know all_cpus so
>> it's faked up with all cpus set for now.
>>
>> I mostly want people to look at the interface in cpuset.h and make sure
>> they agree with it before I start polishing to commit.  I'm fairly happy
>> with the way the syscall api looks now.  The code itself ended up being
>> much more complicated than I'd hoped due to locking considerations.  Try
>> not to look at cpuset_setproc() ;).
>>
>> If you want to actually try the patch, here's a couple of neat things to
>> do with cpuset:
>>
>> cpuset -l 0-4 /bin/sh
>>
>> This creates a new group with a list (-l) of cpus 0-4 inclusive and runs
>> sh in it.
>>
>> cpuset -g -p <sh pid>
>>
>> This will get (-g) the mask of cpus pid (-p) is allowed to run on.
>>
>> cpuset -l 0,2 -p <sh pid>
>>
>> This will restrict sh to running on cpus 0, 2 while its group is still
>> allowed 0-4.
>>
>> cpuset -l 0,2 -c -p <sh pid>
>>
>> This will modify the cpuset (-c) that the sh belongs to.
>>
>> cpuset -l 0-3 -s 1
>>
>> This will modify the set (-s) that all threads are in by default to
>> contain the first 4 cpus leaving the rest idled.
>>
>> Feedback is appreciated.
>>
>
> Jeff,
>
> Is it currently, or will it eventually be possible to assign network threads
> to different cores? Everything appears to be driven by pid, but at least
> according to top all interrupt "processes" show as pid 12. Also, if
> kern.sched.topology returns 0 is it safe to assume I'm not getting the
> benefit of the topology distinction between packages vs cores?

I forgot to remove the topology sysctl.  It is meaningless now.  If your 
machine on boot say something like:
   Cores per package: 2

Then the scheduler is aware of the layout.

As for binding interrupt/kernel threads, you need to get the tid.  Then 
you can set a mask.  er, except I guess I didn't make a -t tid argument 
for cpuset yet.  I'll add that to the next patch.

Thanks,
Jeff

>
> Nick
>