pin/bind a pthread to a processor? (take 2)

[John Giacomoni]

Dan Eischen and Julian Elischer,

I'd really like the ability to "pin" to a processor.  However, I  
think I need something stronger than immediately rescheduling the  
current process as seems to be done in the 4bsd scheduler.  I'd like  
to "own" the cpu as much as possible until the "pinned" thread yields  
it or terminates.


I've run some user-space and kernel-space tests in FreeBSD 6.2 with  
preemption turned off that look as follows:

I created a pthread with both libpthread and libthr that had  
PTHREAD_SCOPE_SYSTEM set with pthread_attr_setscope.  The thread then  
calls a system call module I created that calls sched_bind and  
sched_pin on the thread.

The same is true for the kernel, only I create a thread in the kernel  
with kthread_create and then bind and pin it.

the bound thread in both cases iterates 10,000,000 times, and for  
each iteration
1) takes a timestamp with rdtsc() (x86)
2) spins until the delta from that timestamp is >= 100
3) classifies the delta as short or long, short is a delta <= 400
4) repeat

Short trials are on average 102 cycles (good)
Long trials are on average 950-1100 cycles (bad)

There are 950-1100 long trials for every 10,000,000 trial run.
This means I'm getting an abnormal trial 1300-1550 times a second

While this is not the end of the world, it is causing me difficulties  
with my experiments as I'm trying to process 1,488,095 Ethernet  
frames per second, meaning a new 64B frame is ready every 672ns.   
These abnormalities are causing me to require queuing and reduce the  
maximum processing time to account for it, which is less than ideal :)

Right now I have a setup that lets me stream 100% of frames from an  
input handler thread in the  kernel to a user-space thread and back  
to an output handler kernel thread for all frames >=96B, and 95% of  
the frames for 64B (I can't seem to generate more than that on my  
hardware).

Therefore it looks like I need to find a way to eliminate all non- 
application requested tasks including the periodic scheduler and any  
callout code.


Do either of you have any thoughts or ideas?


I'm attaching the test code I used to generate these numbers.

Thanks in advance for any help!

John Giacomoni

-------------- next part --------------
A non-text attachment was scrubbed...
Name: pintest.tgz
Type: application/octet-stream
Size: 3490 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20070216/0021d050/pintest.obj
-------------- next part --------------

On Feb 9, 2007, at 9:53 PM, Daniel Eischen wrote:

> On Fri, 9 Feb 2007, Julian Elischer wrote:
>
>> Peter Holmes wrote:
>>> This is something I am interested in doing as well. I had  
>>> corresponded with Julian Eischen & Daniel Elischer  about this.
>>
>> Dan Eischen and Julian Elischer :-)
>
> Yes, it needed byte reversal or something.  That would indeed be
> a strange union!
>
> I think it would be nice to have something like Solaris pbind(1)
> and processor_bind(2).
>
> -- 
> DE
> _______________________________________________
> freebsd-hackers at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hackers- 
> unsubscribe at freebsd.org"

--

John.Giacomoni at colorado.edu
University of Colorado at Boulder
Department of Computer Science
Engineering Center, ECCR 1B50
430 UCB
Boulder, CO 80303-0430
USA

-------------- next part --------------