NFS client perf. degradation when SCHED_ULE is used (was when SMP enabled)

[Andriy Gapon]

On 28/05/2017 01:20, Rick Macklem wrote:
> - with the "obvious change" mentioned in r312426's commit message, using
>    (flags & SW_TYPE_MASK) == SWT_RELINQUISH instead of (flag & SWT_RELINQUISH)
>                                                            121minutes

Rick,

can I see how exactly your variant of the obvious change looks in your version?

I am asking, because I meant applying that change to the original code while it
can be also interpreted as applying it to the code after r312426.

That is, does it looks like this:
	preempted = !((td->td_flags & TDF_SLICEEND) || ((flags & SW_TYPE_MASK) ==
SWT_RELINQUISH));

or like this:
	preempted = !(td->td_flags & TDF_SLICEEND) && ((flags & SW_TYPE_MASK) ==
SWT_RELINQUISH);

> I also tested:
>     ((flags & SW_PREEMPT) != 0 || (flags & SW_TYPE_MASK) == SWT_IDLE ||
>       (flags & SW_TYPE_MASK) == SWT_IWAIT)
> and it also resulted in                121minutes

So, this sets the preempted flag for SWT_IDLE and SWT_IWAIT cases.
The flag makes any difference only if the current thread is calling mi_switch()
but remains running (of which typical cases are preemption and yielding).  As
far as I can tell, mi_switch(SWT_IWAIT) is only called when the thread is
already inhibited via TD_SET_IWAIT, so that should not make any difference.
SWT_IDLE is set only when the current thread is an idle thread, so that should
not make any difference either.
Thus, I am puzzled as to why this change could make any difference.

Could you please post full code snippets for each local change that you tried?

Also, could you please capture KTR sched trace while running the test on the
kernel with no local modifications and on the 1yr old kernel?
Ideally, I would like to see the trace with KTR_SCHED | KTR_RUNQ compiled into
the kernel via KTR_COMPILE and then enabled at the run time via
debug.ktr.mask=0x20400000.

Thank you.



-- 
Andriy Gapon