SCHED_ULE should not be the default

[Nathan Whitehorn]

On 12/18/11 04:34, Adrian Chadd wrote:
> The trouble is that there's lots of anecdotal evidence, but noone's
> really gone digging deep into _their_ example of why it's broken. The
> developers who know this stuff don't see anything wrong. That hints to
> me it may be something a little more creepy - as an example, the
> interplay between netisr/swi/taskqueue/callbacks and such. It may be
> that something is being starved that isn't obviously obvious. It's
> just a stab in the dark, but it sounds somewhat plausible based on
> what I've seen ULE do in my network throughput hacking.
>
> I applaud reppie for trying to make it as easy as possible for people
> to use KTR to provide scheduler traces for him to go digging with, so
> please, if you have these issues and you can absolutely reproduce
> them, please follow his instructions and work with him to get him what
> he needs.
>

The thing I've seen is that ULE is substantially more enthusiastic about 
migrating processes between cores than 4BSD. Often, this is a good 
thing, but can increase the rate of cache misses, hurting performance 
for cache-bound processes (I see this particularly in HPC-type 
scientific workloads). It might be interesting to add some kind of 
tunable here.

Another more interesting and slightly longer-term possibility if someone 
wants a project would be to integrate scheduling decisions with hwpmc 
counters, to accumulate statistics on cache hits at each context switch 
and preferentially keep processes with a high hits/misses ratio on the 
same thread/cache domain relative to processes with a low one.
-Nathan

P.S. The other thing that could be very interesting from a research and 
scheduling standpoint would be to integrate heterogeneous SMP support 
into the operating system, with a FreeBSD-4 "Application Processor" 
syscall model. We seem to be going down the road where GPGPU computing 
has MMUs, timer interrupts, IPIs, etc. (the next AMD Fusions, IBM Cell), 
as well as potential systems with both x86 and ARM cores. This is 
something that no operating system currently supports well, and would be 
a place for BSD to shine. If anyone has a free graduate student...