Linux compatible setaffinity.

[Robert Watson]

On Wed, 20 Feb 2008, Jeff Roberson wrote:

>> So perhaps this means a slightly more complex API, but not much more 
>> complex. How about:
>> 
>> int cpuaffinity_get(scope, id, length, mask)
>> int cpuaffinity_getmax(scope, id, length, mask)
>> int cpuaffinity_set(scope, id, length, mask)
>> int cpuaffinity_setmax(scope, id, length, mask)
>> 
>> Scope would be something on the order of process (representing individual 
>> processes or process groups, potentially), id would be the id in that scope 
>> namespace, length and mask would be as you propose.  You could imagine 
>> adding a further field to indicate whether it's the current affinity or the 
>> maximum affinity, but I'm not sure the details matter all that much.  Here 
>> might be some application logic, though:
>
> Well I'm not sure about the max.  How about just a cpuaffinity_get with a 
> scope that specifies what cpus are available to you?  If the set is 
> restricted by a jail or some other mechanism it would be returned in avail. 
> Otherwise all cpus would be returned.  The thread probably wouldn't directly 
> mainpulate its max, rather it would be set by changing the jail or cpu group 
> it belonged to.

I think the actual details don't matter too much as long as we can express 
what we need to, so I'm ok with a special scope for that.  You do raise an 
interesting point, though, on the nature of scope: presumably we'd like to be 
able to query a few different "maximums" in the interest of having a maximally 
debuggable system -- be it the hardware limit, the administrative limit, etc. 
So perhaps the scopes are something more like:

#define	CPUAFF_SCOPE_HARDWARE	1	/* Hardware limits */
#define	CPUAFF_SCOPE_SYSTEM	2	/* System usage mask */
#define	CPUAFF_SCOPE_PROCESS	3	/* Processes and process groups */
#define	CPUAFF_SCOPE_THREAD	4	/* Threads in process */

CPUAFF_SCOPE_HARDWARE would be what the kernel has probed at the physical 
layer.  This would be get-only, and available simply to provide a consistent 
interface when dealing with CPU masks.

CPUAFF_SCOPE_SYSTEM would be what are exposed by the kernel for use by the 
application.  This would be get-only without privilege, and presumably not 
exceed CPUAFF_SCOPE_HARDWARE.  This would be the practical upper bound on what 
a process could use, and would likely be how we implement Jail scoping of 
CPUs.  This might or might not be how one tries to discourage userspace use of 
a particular CPU.

CPUAFF_SCOPE_PROCESS would be the process affinity for the process as a whole. 
It would be get-set without privilege, but limited to CPUAFF_SCOPE_SYSTEM. 
I'm not sure we want that limitation to be something overridden with 
privilege, but I guess we can think about that.

CPUAFF_SCOPE_THREAD would be the thread affinity for an individual thread. 
It would be get-set without privilege.  Do we limit it to CPUAFF_SCOPE_SYSTEM 
or CPUAFF_SCOPE_PROCESS?

Since this involves some access control now, a pondering on access control:

When a process sets the affinity of another thread or process, is it limited 
to the system scope of the current process, or of the target process?  This 
may be a practical question if we're talking about how to deal with a process 
outside of a jail setting the affinity of a process inside jail.  It's easy to 
implement the right thing in the kernel, but does that then imply that the 
system scope query should take a process ID so that a user tool can figure out 
what valid choices are? :-)

I think it's useful to play out a few of these scenarios a bit and see what 
works; we won't be locked into any particular model until it hits a -STABLE 
branch, but getting the underlying primitive right will make it a lot easier 
to implement some of the more mature services we have in mind, making them 
more likely to happen.

Robert N M Watson
Computer Laboratory
University of Cambridge