ixgbe(4) spin lock held too long

[Jason Wolfe]

On Wed, Sep 10, 2014 at 8:24 AM, John Baldwin <jhb at freebsd.org> wrote:

> On Monday, September 08, 2014 03:34:02 PM Eric van Gyzen wrote:
> > On 09/08/2014 15:19, Sean Bruno wrote:
> > > On Mon, 2014-09-08 at 12:09 -0700, Sean Bruno wrote:
> > >> This sort of looks like the hardware failed to respond to us in time?
> > >> Too busy?
> > >>
> > >> sean
> > >
> > > This seems to be affecting my 10/stable machines from 15Aug2014.
> > >
> > > Not a lot of churn in the code so I don't think this is new.  The
> > > afflicted machines, quite a few by my count, appear to have not been
> > > super busy (pushing about 200 Mb/s).
> > >
> > > sean
> > >
> > >> panic: spin lock held too long
> > >>
> > >> GNU gdb 6.1.1 [FreeBSD]
> > >> Copyright 2004 Free Software Foundation, Inc.
> > >> GDB is free software, covered by the GNU General Public License, and
> you
> > >> are
> > >> welcome to change it and/or distribute copies of it under certain
> > >> conditions.
> > >> Type "show copying" to see the conditions.
> > >> There is absolutely no warranty for GDB.  Type "show warranty" for
> > >> details.
> > >> This GDB was configured as "amd64-marcel-freebsd"...
> > >>
> > >> Unread portion of the kernel message buffer:
> > >> spin lock 0xffffffff812a0400 (callout) held by 0xfffff800151fe000 (tid
> > >> 100003) too long
> >
> > TID 100003 is usually a kernel idle thread, which would seem to indicate
> > a dangling lock.  Can you enable WITNESS (without WITNESS_SKIPSPIN) on
> > this box?
>
> Also, do 'tid 100003' and 'bt' in kgdb to see what the thread holding the
> lock
> was doing.
>
> --
> John Baldwin


Sorry for the delay, I've been hoping to catch a crash on one of our
machines running the WITNESS kernel.  Our luck seems to be in short supply,
the machines running sans WITNESS crash in the same manner at a rate of 2/3
a day.  I may have to grow the pool to catch this, but in the meantime here
is the bt/tid.

(kgdb) bt 1000003
#0  0xffffffff80ac39b8 in cpustop_handler () at
/usr/src/sys/amd64/amd64/mp_machdep.c:1432
#1  0xffffffff80ac397f in ipi_nmi_handler () at
/usr/src/sys/amd64/amd64/mp_machdep.c:1417
#2  0xffffffff80ad2d5a in trap (frame=0xffffffff81242830) at
/usr/src/sys/amd64/amd64/trap.c:190
#3  0xffffffff80ab93c3 in nmi_calltrap () at
/usr/src/sys/amd64/amd64/exception.S:505
#4  0xffffffff80734066 in callout_process (now=3278964590047193) at
/usr/src/sys/kern/kern_timeout.c:487
(kgdb) tid 100003
[Switching to thread 40 (Thread 100003)]#0  0xffffffff80ac39b8 in
cpustop_handler () at /usr/src/sys/amd64/amd64/mp_machdep.c:1432
1432            savectx(&stoppcbs[cpu]);