Advice on a multithreaded netisr patch?

[Robert Watson]

On Mon, 6 Apr 2009, Barney Cordoba wrote:

> Is there a way to give a kernel thread exclusive use of a core? I know you 
> can pin a kernel thread with sched_bind(), but is there a way to keep other 
> threads from using the core? On an 8 core system it almost seems that the 
> randomness of more cores is a negative in some situations.
>
> Also, I've noticed that calling sched_bind() during bootup is a bad thing in 
> that it locks the system. I'm not certain but I suspect its the thread_lock 
> that is the culprit. Is there a clean way to determine that its safe to lock 
> curthread and do a cpu bind?

There isn't an interface to cleanly express "Use CPUs 4-7 for only network 
processing".  You can configure the system this way using the cpuset command 
(including directing the low-level interrupts to specific CPUs in 8.x), but if 
we think this is going to be a frequently desired policy, a bit more 
abstraction will be required.

I'm not familiar with the problem you're seeing with sched_bind() -- I'm using 
it from within some of my code without a problem, and that's fairly early in 
the boot.  A number of deadlocks are possible if one isn't very careful early 
in the boot though, so I might look specifically for some of those: if you 
migrate a thread to a CPU that isn't yet started, it won't be able to run 
until the CPU has started.  This means it's important not to migrate threads 
that might lead to priority version-like deadlocks:

- Be careful not to migrate threads that hold locks the system requires to get
   to the point where multiple CPUs run.
- Be careful not to migrate threads that will signal a resource being
   available, such as a device driver, required to get to the point where
   multiple CPUs run.
- Be careful not to migrate the main boot thread.

Could you be running into one of those cases?  Usually they're fairly easy to 
diagnose using DDB, if you can get into it, because you can see what the main 
boot thread is waiting for, and reason about what's holding it.  Are you able 
to get into DDB when this occurs?  (Perhaps using an NMI?)

Robert N M Watson
Computer Laboratory
University of Cambridge