panic: refcount inconsistency: found: 0 total: 1

Tue Nov 3 14:04:40 UTC 2015

This was on my laptop; yesterday, it built & booted:

FreeBSD g1-252.catwhisker.org 11.0-CURRENT FreeBSD 11.0-CURRENT #230  r290270M/290270:1100085: Mon Nov  2 05:03:07 PST 2015     root at g1-252.catwhisker.org:/common/S4/obj/usr/src/sys/CANARY  amd64

OK; today, after building:

FreeBSD localhost 11.0-CURRENT FreeBSD 11.0-CURRENT #231  r290334M/290334:1100086: Tue Nov  3 04:51:24 PST 2015     root at g1-252.catwhisker.org:/common/S4/obj/usr/src/sys/CANARY  amd64

I tried booting it, and during the transition to multi-user mode,
once ipfw was being invoked, I got the above-cited panic.  Circumvention
was to leave it disconnected from a network (turn off the WiFi
switch, in my case), so we don't get a chance to use the network.

I was able to get a dump by explicitly typing "call doadump" -- an
earlier attempt at "panic" didn't capture one.  Stack trace:

#0  doadump (textdump=0) at pcpu.h:221
221     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) #0  doadump (textdump=0) at pcpu.h:221
#1  0xffffffff8037b6b6 in db_fncall (dummy1=<value optimized out>, 
    dummy2=<value optimized out>, dummy3=<value optimized out>, 
    dummy4=<value optimized out>) at /usr/src/sys/ddb/db_command.c:568
#2  0xffffffff8037b14e in db_command (cmd_table=0x0)
    at /usr/src/sys/ddb/db_command.c:440
#3  0xffffffff8037aee4 in db_command_loop ()
    at /usr/src/sys/ddb/db_command.c:493
#4  0xffffffff8037d97b in db_trap (type=<value optimized out>, code=0)
    at /usr/src/sys/ddb/db_main.c:251
#5  0xffffffff80a270f3 in kdb_trap (type=3, code=0, tf=<value optimized out>)
    at /usr/src/sys/kern/subr_kdb.c:654
#6  0xffffffff80db6668 in trap (frame=0xfffffe060bdde1d0)
    at /usr/src/sys/amd64/amd64/trap.c:549
#7  0xffffffff80d961f7 in calltrap ()
    at /usr/src/sys/amd64/amd64/exception.S:234
#8  0xffffffff80a267db in kdb_enter (why=0xffffffff812a5566 "panic", 
    msg=0x80 <Address 0x80 out of bounds>) at cpufunc.h:63
#9  0xffffffff809ea01f in vpanic (fmt=<value optimized out>, 
    ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:750
#10 0xffffffff809e9e76 in kassert_panic (fmt=<value optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:647
#11 0xffffffff80c2a788 in ipfw_rewrite_rule_uidx (chain=0xffffffff81be5310, 
    ci=0xfffffe060bdde4b8) at /usr/src/sys/netpfil/ipfw/ip_fw_table.c:3395
#12 0xffffffff80c267c3 in commit_rules (chain=0xffffffff81be5310, 
    rci=0xfffffe060bdde4b8, count=1)
    at /usr/src/sys/netpfil/ipfw/ip_fw_sockopt.c:678
#13 0xffffffff80c25d80 in add_rules (chain=0xffffffff81be5310, 
    op3=<value optimized out>, sd=<value optimized out>)
    at /usr/src/sys/netpfil/ipfw/ip_fw_sockopt.c:2594
#14 0xffffffff80c232f4 in ipfw_ctl3 (sopt=0xfffffe060bdde920)
    at /usr/src/sys/netpfil/ipfw/ip_fw_sockopt.c:3242
#15 0xffffffff80b3d8b1 in rip_ctloutput (so=<value optimized out>, 
    sopt=0xfffffe060bdde920) at /usr/src/sys/netinet/raw_ip.c:588
#16 0xffffffff80a72bc6 in sogetopt (so=0xfffff80009e658b8, 
    sopt=0xfffffe060bdde920) at /usr/src/sys/kern/uipc_socket.c:2731
#17 0xffffffff80a7729e in kern_getsockopt (td=0xfffff800098119a0, 
    s=<value optimized out>, level=<value optimized out>, 
    name=<value optimized out>, val=<value optimized out>, valseg=464, 
    valsize=0xfffffe060bdde98c) at /usr/src/sys/kern/uipc_syscalls.c:1540
#18 0xffffffff80a771a0 in sys_getsockopt (td=0xfffff800098119a0, 
    uap=0xfffffe060bddea40) at /usr/src/sys/kern/uipc_syscalls.c:1486
#19 0xffffffff80db7519 in amd64_syscall (td=0xfffff800098119a0, traced=0)
    at subr_syscall.c:140
#20 0xffffffff80d964db in Xfast_syscall ()
    at /usr/src/sys/amd64/amd64/exception.S:394
#21 0x0000000800b2cbea in ?? ()
Previous frame inner to this frame (corrupt stack?)
Current language:  auto; currently minimal
(kgdb) 

I've copied the vmcore.z & core.txt.7 to
<http://www.catwhisker.org/~david/FreeBSD/head/ipfw/>; gzipped
copies are also available:

                    Index of /~david/FreeBSD/head/ipfw

 Icon   Name                    Last modified      Size  Description
  _____________________________________________________________________
 [PARENTDIR]  Parent Directory                             -
 [TXT]  core.txt.7              2015-11-03 05:22  155K
 [   ]  core.txt.7.gz           2015-11-03 05:22   35K
 [   ]  vmcore.7                2015-11-03 05:22  528M
 [   ]  vmcore.7.gz             2015-11-03 05:22   45M
  _____________________________________________________________________

I'll start taking a closer look at recent changes (e.g., in
src/sys/netpfil/ipfw), but I'm not really all that familiar with
the code.

Peace,
david
-- 
David H. Wolfskill				david at catwhisker.org
Those who would murder in the name of God or prophet are blasphemous cowards.

See http://www.catwhisker.org/~david/publickey.gpg for my public key.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 949 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-current/attachments/20151103/45b60e15/attachment.bin>