ULE steal_idle questions

[Andriy Gapon]

On 23/08/2017 18:04, Don Lewis wrote:
> I've been looking at the steal_idle code in tdq_idled() and found some
> things that puzzle me.
> 
> Consider a machine with three CPUs:
>   A, which is idle
>   B, which is busy running a thread
>   C, which is busy running a thread and has another thread in queue
> It would seem to make sense that the tdq_load values for these three
> CPUs would be 0, 1, and 2 respectively in order to select the best CPU
> to run a new thread.
> 
> If so, then why do we pass thresh=1 to sched_highest() in the code that
> implements steal_idle?  That value is used to set cs_limit which is used
> in this comparison in cpu_search:
>                         if (match & CPU_SEARCH_HIGHEST)
>                                 if (tdq->tdq_load >= hgroup.cs_limit &&
> That would seem to make CPU B a candidate for stealing a thread from.
> Ignoring CPU C for the moment, that shouldn't happen if the thread is
> running, but even if it was possible, it would just make CPU B go idle,
> which isn't terribly helpful in terms of load balancing and would just
> thrash the caches.  The same comparison is repeated in tdq_idled() after
> a candidate CPU has been chosen:
>                 if (steal->tdq_load < thresh || steal->tdq_transferable == 0) {
>                         tdq_unlock_pair(tdq, steal);
>                         continue;
>                 }
> 
> It looks to me like there is an off-by-one error here, and there is a
> similar problem in the code that implements kern.sched.balance.


I agree with your analysis.  I had the same questions as well.
I think that the tdq_transferable check is what saves the code from
running into any problems.  But it indeed would make sense for the code
to understand that tdq_load includes a currently running, never
transferable thread as well.

> The reason I ask is that I've been debugging random segfaults and other
> strange errors on my Ryzen machine and the problems mostly go away if I
> either disable kern.sched.steal_idle and kern_sched.balance, or if I
> leave kern_sched.steal_idle enabled and hack the code to change the
> value of thresh from 1 to 2.  See
> <https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=221029> for the gory
> details.  I don't know if my CPU has what AMD calls the "performance
> marginality issue".

I have been following your experiments and it's interesting that
"massaging" the CPU in certain ways makes it a bit happier.  But
certainly the fault is with the CPU as the code is trouble-free on many
different architectures including x86, and various processors from both
Intel and AMD [with earlier CPU families].


-- 
Andriy Gapon