RELENG_7 crash

[Ruslan Ermilov]

On Tue, Apr 21, 2009 at 11:20:13AM -0400, Mike Tancsa wrote:
> At 11:11 AM 4/21/2009, John Baldwin wrote:
> 
> >Can you do 'frame 7' followed by 'l', 'p ifp', and 'p ifp->if_snd'?
> 
> Hi,
> 
> 
> kgdb) bt
> #0  doadump () at pcpu.h:196
> #1  0xc05964d7 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418
> #2  0xc05967a9 in panic (fmt=Variable "fmt" is not available.
> ) at /usr/src/sys/kern/kern_shutdown.c:574
> #3  0xc07f64ac in trap_fatal (frame=0xe766ea6c, eva=104) at 
> /usr/src/sys/i386/i386/trap.c:939
> #4  0xc07f6730 in trap_pfault (frame=0xe766ea6c, usermode=0, eva=104) 
> at /usr/src/sys/i386/i386/trap.c:852
> #5  0xc07f70dc in trap (frame=0xe766ea6c) at /usr/src/sys/i386/i386/trap.c:530
> #6  0xc07db7eb in calltrap () at /usr/src/sys/i386/i386/exception.s:159
> #7  0xc0637146 in sysctl_ifdata (oidp=0xc08816a0, arg1=0xe766ec24, 
> arg2=2, req=0xe766eba4) at /usr/src/sys/net/if_mib.c:127
> #8  0xc059fd77 in sysctl_root (oidp=Variable "oidp" is not available.
> ) at /usr/src/sys/kern/kern_sysctl.c:1413
> #9  0xc059ff14 in userland_sysctl (td=0xc5374460, name=0xe766ec14, 
> namelen=6, old=0x0, oldlenp=0xbfbf8478, inkernel=0, new=0x0,
>      newlen=0, retval=0xe766ec10, flags=0) at 
> /usr/src/sys/kern/kern_sysctl.c:1506
> #10 0xc05a0064 in __sysctl (td=0xc5374460, uap=0xe766ecfc) at 
> /usr/src/sys/kern/kern_sysctl.c:1443
> #11 0xc07f6a85 in syscall (frame=0xe766ed38) at 
> /usr/src/sys/i386/i386/trap.c:1090
> #12 0xc07db850 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:255
> #13 0x00000033 in ?? ()
> Previous frame inner to this frame (corrupt stack?)
> (kgdb) frame 7
> #7  0xc0637146 in sysctl_ifdata (oidp=0xc08816a0, arg1=0xe766ec24, 
> arg2=2, req=0xe766eba4) at /usr/src/sys/net/if_mib.c:127
> 127                     ifp->if_snd.ifq_drops = ifmd.ifmd_snd_drops;
> (kgdb) l
> 122                     DONTCOPY(baudrate);
> 123     #undef DONTCOPY
> 124     #define COPY(fld) ifp->if_##fld = ifmd.ifmd_##fld
> 125                     COPY(data);
> 126                     ifp->if_snd.ifq_maxlen = ifmd.ifmd_snd_maxlen;
> 127                     ifp->if_snd.ifq_drops = ifmd.ifmd_snd_drops;
> 128     #undef COPY
> 129                     break;
> 130
> 131             case IFDATA_LINKSPECIFIC:
> (kgdb) p ifp
> $1 = (struct ifnet *) 0x0
> (kgdb) p ifp->if_snd
> Cannot access memory at address 0xf4
> (kgdb)
> 
> 
> Is it possible I am running into some of the interface lock fixes 
> rwatson has been working on ?  This box has a lot of ng interfaces 
> which come and go. Perhaps snmp asking about an interface that just 
> went away caused the panic ? I disabled bsnmp since the reboot and 
> the box has been up for 10hrs so far.
> 
It's a documented bug:

: revision 1.281
: date: 2008/06/26 23:05:28;  author: rwatson;  state: Exp;  lines: +69 -12
: SVN rev 180042 on 2008-06-26 23:05:28Z by rwatson
: 
: Introduce locking around use of ifindex_table, whose use was previously
: unsynchronized.  While races were extremely rare, we've now had a
: couple of reports of panics in environments involving large numbers of
: IPSEC tunnels being added very quickly on an active system.
: 
: - Add accessor functions ifnet_byindex(), ifaddr_byindex(),
:   ifdev_byindex() to replace existing accessor macros.  These functions
:   now acquire the ifnet lock before derefencing the table.
: - Add IFNET_WLOCK_ASSERT().
: - Add static accessor functions ifnet_setbyindex(), ifdev_setbyindex(),
:   which set values in the table either asserting of acquiring the ifnet
:   lock.
: - Use accessor functions throughout if.c to modify and read
:   ifindex_table.
: - Rework ifnet attach/detach to lock around ifindex_table modification.
: 
: Note that these changes simply close races around use of ifindex_table,
: and make no attempt to solve the probem of disappearing ifnets.  Further
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
: refinement of this work, including with respect to ifindex_table
: resizing, is still required.
: 
: In a future change, the ifnet lock should be converted from a mutex to an
: rwlock in order to reduce contention.
: 
: Reviewed and tested by:	brooks


Cheers,
-- 
Ruslan Ermilov
ru at FreeBSD.org
FreeBSD committer