Is kern.sched.preempt_thresh=0 a sensible default?

[Steve Kargl]

On Sat, Jun 09, 2018 at 06:07:15PM -0700, Don Lewis wrote:
> On  9 Jun, Stefan Esser wrote:
> 
> > 3) Programs that evenly split the load on all available cores have been
> >    suffering from sub-optimal assignment of threads to cores. E.g. on a
> >    CPU with 8 (virtual) cores, this resulted in 6 cores running the load
> >    in nominal time, 1 core taking twice as long because 2 threads were
> >    scheduled to run on it, while 1 core was mostly idle. Even if the
> >    load was initially evenly distributed, a woken up process that ran on
> >    one core destroyed the symmetry and it was not recovered. (This was a
> >    problem e.g. for parallel programs using MPI or the like.)
> 
> When a core is about to go idle or first enters the idle state it will
> search for the most heavily loaded core and steal a thread from it.  The
> core will only go to sleep if it can't find a non-running thread to
> steal.
> 
> If there are N cores and N+1 runnable threads, there is a long term load
> balancer than runs periodically.  It searches for the most and least
> loaded cores and moves a thread from the former to the latter.  That
> prevents the same pair of threads from having to share the same core
> indefinitely.
> 
> There is an observed bug where a low priority thread can get pinned to a
> particular core that is already occupied by a high-priority CPU-bound
> thread that never releases the CPU.  The low priority thread can't
> migrate to another core that subsequently becomes available because it
> it is pinned.  It is not known how the thread originally got into this
> state.  I don't see any reason for 4BSD to be immune to this problem.
> 

It is a well-known problem that an over-subscribed ULE kernel
has much worse performance than a 4BSD kernel.  I've posted
more than once with benchmark numbers that demonstrate the problem.

-- 
Steve