insanely-high interrupt rates -- PARTIAL resolution (Pi2)

[Ian Lepore]

On Sun, 2019-04-21 at 12:00 -0600, Warner Losh wrote:
> On Sun, Apr 21, 2019 at 11:58 AM Ian Lepore <ian at freebsd.org> wrote:
> 
> > On Wed, 2019-04-17 at 14:56 -0500, Karl Denninger wrote:
> > > On 4/9/2019 19:25, Ian Lepore wrote:
> > > > On Tue, 2019-04-09 at 09:55 -0500, Karl Denninger wrote:
> > > > > On 4/3/2019 11:48, Andrew Gierth wrote:
> > > > > > [...]
> > > > 
> > > > I've just posted https://reviews.freebsd.org/D19871 for this.
> > > > Hopefully I'll get it committed in a day or so and merged to
> > > > 12-
> > > > stable
> > > > a few days after that.
> > > > 
> > > > -- Ian
> > > 
> > > I am running that now on a Pi2 and so far the load problem is
> > > gone
> > > but
> > > the spurious interrupt warnings are not....
> > > 
> > > 
> > 
> > [...]
> > > 
> > > On my bench without the I2c inputs connected (which do analog
> > > reads) I
> > > do NOT get the spurious interrupt prints.  With it connected I
> > > do.  The
> > > process that reads them is code that is running in both cases,
> > > but if it
> > > cannot find the I2c devices it logs the error but continues, so
> > > all it
> > > gets to is trying to open the unit, doesn't see it when probed,
> > > and
> > > gives up.
> > > 
> > > It appears that I2c is an inherent part of the spurious interrupt
> > > thing
> > > still and while the timer issue appears to be fixed that doesn't
> > > resolve
> > > the other problem.
> > > 
> > > Any ideas on how to track down exactly what is generating those
> > > warnings?
> > > 
> > > 
> > 
> > After spending the whole day yesterday trying all the usual driver
> > techniques for eliminating spurious interrupts, I was unable to
> > make
> > them go away completely, but I also convinced myself they're
> > harmless.
> > 
> > I was a little surprised that the "read after write" technique
> > didn't
> > work.  That is, after writing to the i2c control register to clear
> > all
> > the interrupt-enable bits, read back that register.  In theory, at
> > least on normal arm chips, that ensures that the prior write has
> > reached the hardware before the read can procede, so it's a way to
> > guarantee that the write has taken effect and the interrupt can no
> > longer be asserted, before returning from the interrupt
> > handler.  But,
> > on the rpi chips even that doesn't work... you can read back the
> > register and verify the interrupt-enable bits are cleared, and
> > still
> > after returning from the handler, it re-interrupts immediately.
> > 
> > If you stick in a nice long DELAY() after clearing the control
> > register, the spurious interrupts go away, but that's a horrible
> > fix.
> > It would be especially horrible for i2c devices that do a lot of
> > transfers, you'd end up with the delay time overwhelming the time
> > to do
> > the actual transfers themselves.
> > 
> > So, in r346489, I moved the reporting of the spurious transfers
> > under
> > the bootverbose flag, so that normally you just won't see them
> > anymore,
> > but we can still enable the reporting if we suspect some device
> > driver
> > is behaving badly.  I'll mfc that change to 12-stable after a few
> > days.
> > 
> 
> vmstat -i will also show if you're system has an unusually high interrupt
> rate in general as well, and is preferable to spamming the console with
> printfs :)
> 
> Warner

vmstat doesn't report spurious interrupts in any way, though.  I
considered making it do so as one of the possible fixes here, but it
turns out to be complicated... we need to do a bit of reworking of the
INTRNG code as it related to interrupt counts.  For example, on x86 you
get this from vmstat -i:

cpu0:timer                      42006521         80
cpu1:timer                      32510560         62

But on arm, all timer interrupts are counted as belonging to the
generic_timer0 device.  When I tried to figure out how to split that
into per-cpu reporting like x86 does, I discovered what a mess the
intrstats stuff in INTRNG is right now.  So, a project for another day,
I guess.

-- Ian