Linux compatible setaffinity.

[Robert Watson]

On Thu, 20 Dec 2007, Andre Oppermann wrote:

> Not that I'm against a Linux compatible sched_setaffinity(), it's just not 
> as practical to use as other constructs.

Another thought I shared with Jeff earlier today was that we'd like to be able 
to represent some more systemic ideas, such as:

- This jail is able to run on logical CPUs 7 and 8, whereas that jail can use
   9, 10, and 11.  Scope affinity management in some form to those CPUs.

- The tasks on CPUs 4, 5, 6, and 7 should be relocated to CPUs 8, 9, 10, and
   11 so that those former CPUs can be taken offline.

- I don't care what physical die my threads run on, but I want them all to be
   on the same die.

At least the first of these has immediate applicability, and might be 
represented as an additional per-thread inherited mask.  However, it won't 
help with the second, which implies a level of indirection between the 
requested medium and the assigned medium.  Solaris has a notion of CPU set, 
which I'm not sure we want verbatim, but might consider whether it has some 
useful properties.  In particular, it would allow you to perform certain types 
of assignment operations more abstractly, such as "This jail runs on CPU set 
X; the customer has paid more money, so add another two CPUs to the set and 
have the jail automatically expand to include them", which isn't 
well-represented if you have to walk lots and lots of thread and process 
masks.

I'm not opposed to simply getting the Linux API in the tree for now, but I'd 
rather it weren't MFC'd until we've thought a bit more about whether we need a 
stronger abstraction.  One thing to think about is whether we couldn't layer 
the Linux API over another abstraction -- i.e., have the API's notion of CPU 
number be with respect to the current CPU set, so that if the CPU set is 
migrated, the relative assignments and pinnings still apply, just to different 
CPUs.

Finally, one of the things we will want to start addressing in 8 is affinity 
for TCP connections.  Kip and I have both been waving our hands at some stuff 
in this area, but we'll want to formalize it.  Having a clear idea of our 
thread/process affinity model will be an important part of understanding what 
we're going to do for network connections, because presumably we'll want the 
affinities of kernel objects to be related in some way to the affinities 
expressed by user threads and processes...  I've done a bit of hacking 
regarding simply allowing user threads to request a specific affinity for a 
TCP connection, but that only makes sense in the context of the new TCP 
locking work.  However, if requests for affinity are with respect to logical 
CPUs within a CPU set, the expression and maintenance of that affinity for the 
TCP connection becomes quite different.

Robert N M Watson
Computer Laboratory
University of Cambridge