Peformance issues with r278325

[Stanislav Sedov]

> On Mar 18, 2016, at 10:37 AM, John Baldwin <jhb at freebsd.org> wrote:
> 
> On Thursday, March 17, 2016 10:45:39 AM Adrian Chadd wrote:
>> On 17 March 2016 at 10:23, John Baldwin <jhb at freebsd.org> wrote:
>> 
>>> 
>>> I had not expected this commit to have this impact, but Konstantin is correct.
>>> I wonder if simply reducing the DELAY() from 5 down to 1 or so would be
>>> sufficient?  (Note that you need to adjust the prior loop to use += 1 instead
>>> of += 5 in that case.)
>>> 
>>> Note that DETECT_DEADLOCK is not enabled by default, so the AFTER_SPIN case
>>> (which waits for an IPI just sent to be delivered) shouldn't be enabled (and
>>> in fact I'd like to just remove that code entirely).  This means that only
>>> BEFORE_SPIN should be spinning, and it should only be spinning if a CPU sends
>>> IPIs back to back such that the previous IPI is still pending (not yet
>>> delivered) when the CPU wants to send another IPI.
>>> 
>>> We can probably assume a TSC if we have SMP, so if changing the delay from 5
>>> to 1 doesn't work we can try just using the TSC directly to control the
>>> spin length and go back to using a simple pause.
>>> 
>>> I have an old set of changes that might also be interesting that permit
>>> TLB shootdown IPI handlers to run while spinlocks are held (by using cr8/TPR
>>> to control interrupts when a spinlock is held instead of disabling all
>>> interrupts).  I haven't found a workload where that helped yet.  However,
>>> yours might be an interesting workload to try those changes out on.
>> 
>> Do you think it's worth just reverting it for now just so it lands in
>> 10.3-RELEASE?
> 
> Probably.  If the '1' change fixes it that is a simple test, otherwise we
> can revert in 10.x.  I think I'll likely just convert it to use a direct
> TSC delay loop always in HEAD (assuming that verifies ok in testing as well).
> 

FWIW we are currently testing the delay '1' change.  Unfortunately, the test is
not easy to repeat (we didn't find a synthetic one yet that results in the same
outcome), so it does take more time that I would like.  Will follow up with the 
results.

We did try HEAD as well a while ago, and although it exhibited the same pattern.
However it did not utilize the x2apic unfortunately, as it does seem to be disables
in the BIOS (FreeBSD reports it being disables in the DMAR table).

Thanks for looking into it!

--
Stanislav Sedov
ST4096-RIPE