Peformance issues with r278325

[John Baldwin]

On Thursday, March 17, 2016 10:45:39 AM Adrian Chadd wrote:
> On 17 March 2016 at 10:23, John Baldwin <jhb at freebsd.org> wrote:
> 
> >
> > I had not expected this commit to have this impact, but Konstantin is correct.
> > I wonder if simply reducing the DELAY() from 5 down to 1 or so would be
> > sufficient?  (Note that you need to adjust the prior loop to use += 1 instead
> > of += 5 in that case.)
> >
> > Note that DETECT_DEADLOCK is not enabled by default, so the AFTER_SPIN case
> > (which waits for an IPI just sent to be delivered) shouldn't be enabled (and
> > in fact I'd like to just remove that code entirely).  This means that only
> > BEFORE_SPIN should be spinning, and it should only be spinning if a CPU sends
> > IPIs back to back such that the previous IPI is still pending (not yet
> > delivered) when the CPU wants to send another IPI.
> >
> > We can probably assume a TSC if we have SMP, so if changing the delay from 5
> > to 1 doesn't work we can try just using the TSC directly to control the
> > spin length and go back to using a simple pause.
> >
> > I have an old set of changes that might also be interesting that permit
> > TLB shootdown IPI handlers to run while spinlocks are held (by using cr8/TPR
> > to control interrupts when a spinlock is held instead of disabling all
> > interrupts).  I haven't found a workload where that helped yet.  However,
> > yours might be an interesting workload to try those changes out on.
> 
> Do you think it's worth just reverting it for now just so it lands in
> 10.3-RELEASE?

Probably.  If the '1' change fixes it that is a simple test, otherwise we
can revert in 10.x.  I think I'll likely just convert it to use a direct
TSC delay loop always in HEAD (assuming that verifies ok in testing as well).

-- 
John Baldwin