Linux compatible setaffinity.

[Jeff Roberson]

On Sat, 12 Jan 2008, Jeff Roberson wrote:

> On Sat, 12 Jan 2008, Daniel Eischen wrote:
>
>> On Sat, 12 Jan 2008, Jeff Roberson wrote:
>> 
>>> Now, there is one problem with the linux api that I want to discuss before 
>>> I commit it.  The current patch always works on curthread.  However, the 
>>> api allows for setting the binding of a pid.  I believe, although I'm not 
>>> certain, that pids and tids in linux are in the same number space.  It's 
>>> not clear to me whether you can set an affinity for an entire process and 
>>> have it effect an individual thread or whether you set it on a thread by 
>>> thread basis.  When supplying a non-curproc pid do you bind all threads in 
>>> the target process?
>>> 
>>> Are our tids and pids in the same number space?  And are they available to 
>>> application programmers?  I haven't followed that very carefully.
>> 
>> I believe marcel made tids and pids disjoint so that any pid is
>> never equal to any tid.  But regardless, I don't think we want
>> to rely on that.  I would prefer the Solaris approach of specifying
>> what we want (pid, tid, jail id, etc) as an argument in the API
>> so there is no confusion.
>
> Yes, I would prefer that as well I believe.  So I'll add an extra parameter 
> and in the linux code we'll use whatever their default is.  Of course the 
> initial implementation will still only support curthread but I plan on 
> finishing the rest before 8.0 is done.

So what does everyone think of something like this:

int cpuaffinity(int cmd, long which, int masksize, unsigned *mask);

#define AFFINITY_GET	0x1
#define	AFFINITY_SET	0x2
#define	AFFINITY_PID	0x4
#define	AFFINITY_TID	0x8

I'm not married to any of these names.  If you know of something that 
would be more regular please comment.

Behavior according to flags would be as such:

Get or set affinity and fetch from or store into mask.  Error if mask is 
not large enough.  Fill with zeros if it's too large.

If pid is specified on set all threads in the pid are set to the requested 
affinity.  On get it doesn't make much sense but I guess I'll make it the 
union of all threads affinities.

If tid is specified the mask applies only to the requested tid.

The mask is always inherited from the creating thread and propagates on 
fork().

I have these semantics implemented and appearing to work in ULE.  I can 
implement them in 4BSD but it will be very inefficient in some edge cases 
since each cpu doesn't have its own run queue.

Binding and pinning are still both supported via the same kernel 
interfaces as they were.  They are considered to override user specified 
affinity.  This means the kernel can temporarily bind  a thread to a cpu 
that it does not have affinity for.  I may add an assert to verify that 
we never leave the kernel with binding still set so userspace sees only 
the cpus it requests.

The thread's affinity is stored in a cpumask variable in the thread 
structure.  If someone wanted to implement restricting a jail to a 
particular cpu they could add an affinity cmd that would walk all 
processes belonging to a jail and restrict their masks appropriately. 
You'd also want to check a jail mask on each call to affinity().

Linux sched_setaffinity() should be a subset of this functionality and 
thus easily support.

Comments appreciated.  This will go in late next week.

Thanks,
Jeff

>
> Jeff
>
>> 
>> -- 
>> DE
>> 
> _______________________________________________
> freebsd-arch at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-arch
> To unsubscribe, send any mail to "freebsd-arch-unsubscribe at freebsd.org"
>