kern/153938: [run] [panic] [patch] Workaround for use-after-free
panic
Juergen Lock
nox at jelal.kn-bremen.de
Fri Jan 21 18:30:14 UTC 2011
The following reply was made to PR kern/153938; it has been noted by GNATS.
From: Juergen Lock <nox at jelal.kn-bremen.de>
To: PseudoCylon <moonlightakkiy at yahoo.ca>
Cc: bug-followup at freebsd.org, Juergen Lock <nox at jelal.kn-bremen.de>
Subject: Re: kern/153938: [run] [panic] [patch] Workaround for use-after-free
panic
Date: Fri, 21 Jan 2011 19:21:20 +0100
On Thu, Jan 20, 2011 at 04:35:48PM -0800, PseudoCylon wrote:
> Hello,
>
> I have applied changes. Please check it out.
> http://gitorious.org/run/run/trees/ratectl_fix/dev/usb/wlan
I added debug output again and then after a while got a deadlock [1]
that I suspect is caused by a lor, see below. (lock order reversal
between "run0" and "run0_node_lock" i.e. RUN_LOCK and IEEE80211_NODE_LOCK.)
It's possible this was triggered by the first DPRINTFN() in
run_node_cleanup() (that I turned into a device_printf() and meanwhile
have disabled, maybe it caused a taskswitch) - but in any case I'd
say this is not safe i.e. needs to be fixed. :)
[1] box stayed up but several things got stuck so in the end I had
to drop to ddb and do a `call doadump', and fortunately this time
the dump worked too...
(kgdb) info threads
[...]
at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850
121 Thread 100418 (PID=31634: hostapd) sched_switch (
td=0xffffff00758633e0, newtd=0xffffff0005b40000, flags=Variable "flags" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850
[...]
72 Thread 100064 (PID=14: usb/usbus6) sched_switch (td=0xffffff0005c21000,
newtd=0xffffff0005c20ba0, flags=Variable "flags" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850
71 Thread 100063 (PID=14: usb/usbus6) sched_switch (td=0xffffff0005c213e0,
newtd=0xffffff00018837c0, flags=Variable "flags" is not available.
)
---Type <return> to continue, or q <return> to quit---
at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850
70 Thread 100062 (PID=14: usb/usbus6) sched_switch (td=0xffffff0005c217c0,
newtd=0xffffff0005c213e0, flags=Variable "flags" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850
69 Thread 100061 (PID=14: usb/usbus6) sched_switch (td=0xffffff0005c21ba0,
newtd=0xffffff0005c217c0, flags=Variable "flags" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850
68 Thread 100057 (PID=14: usb/usbus5) sched_switch (td=0xffffff0005c25ba0,
newtd=0xffffff00018907c0, flags=Variable "flags" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850
67 Thread 100056 (PID=14: usb/usbus5) sched_switch (td=0xffffff0005a853e0,
newtd=0xffffff00018833e0, flags=Variable "flags" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850
66 Thread 100055 (PID=14: usb/usbus5) sched_switch (td=0xffffff0005a857c0,
newtd=0xffffff00018907c0, flags=Variable "flags" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850
65 Thread 100054 (PID=14: usb/usbus5) sched_switch (td=0xffffff0005a85ba0,
---Type <return> to continue, or q <return> to quit---
newtd=0xffffff0005a857c0, flags=Variable "flags" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850
64 Thread 100052 (PID=14: usb/usbus4) sched_switch (td=0xffffff0005b403e0,
newtd=0xffffff0005a85ba0, flags=Variable "flags" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850
63 Thread 100051 (PID=14: usb/usbus4) sched_switch (td=0xffffff0005b407c0,
newtd=0xffffff00018833e0, flags=Variable "flags" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850
62 Thread 100050 (PID=14: usb/usbus4) sched_switch (td=0xffffff0005b40ba0,
newtd=0xffffff0005b407c0, flags=Variable "flags" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850
61 Thread 100049 (PID=14: usb/usbus4) sched_switch (td=0xffffff0005b41000,
newtd=0xffffff0005b40ba0, flags=Variable "flags" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850
60 Thread 100048 (PID=14: usb/usbus3) sched_switch (td=0xffffff0005b413e0,
newtd=0xffffff0005b41000, flags=Variable "flags" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850
---Type <return> to continue, or q <return> to quit---
59 Thread 100047 (PID=14: usb/usbus3) sched_switch (td=0xffffff0005b417c0,
newtd=0xffffff0001883000, flags=Variable "flags" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850
58 Thread 100046 (PID=14: usb/usbus3) sched_switch (td=0xffffff0005b41ba0,
newtd=0xffffff0005b417c0, flags=Variable "flags" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850
57 Thread 100045 (PID=14: usb/usbus3) sched_switch (td=0xffffff0001a2cba0,
newtd=0xffffff0005b41ba0, flags=Variable "flags" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850
56 Thread 100043 (PID=14: usb/usbus2) sched_switch (td=0xffffff0005a813e0,
newtd=0xffffff00018837c0, flags=Variable "flags" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850
55 Thread 100042 (PID=14: usb/usbus2) sched_switch (td=0xffffff0005a817c0,
newtd=0xffffff00018907c0, flags=Variable "flags" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850
54 Thread 100041 (PID=14: usb/usbus2) sched_switch (td=0xffffff0005a81ba0,
newtd=0xffffff0001883000, flags=Variable "flags" is not available.
)
---Type <return> to continue, or q <return> to quit---
at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850
53 Thread 100040 (PID=14: usb/usbus2) sched_switch (td=0xffffff0005a83000,
newtd=0xffffff0005a81ba0, flags=Variable "flags" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850
52 Thread 100039 (PID=14: usb/usbus1) sched_switch (td=0xffffff0005a833e0,
newtd=0xffffff00018907c0, flags=Variable "flags" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850
51 Thread 100038 (PID=14: usb/usbus1) sched_switch (td=0xffffff0005a837c0,
newtd=0xffffff00018837c0, flags=Variable "flags" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850
50 Thread 100037 (PID=14: usb/usbus1) sched_switch (td=0xffffff0005a83ba0,
newtd=0xffffff0005a837c0, flags=Variable "flags" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850
49 Thread 100036 (PID=14: usb/usbus1) sched_switch (td=0xffffff0005a85000,
newtd=0xffffff0005a83ba0, flags=Variable "flags" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850
48 Thread 100035 (PID=14: usb/usbus0) sched_switch (td=0xffffff00019fe7c0,
---Type <return> to continue, or q <return> to quit---
newtd=0xffffff0005a85000, flags=Variable "flags" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850
47 Thread 100034 (PID=14: usb/usbus0) sched_switch (td=0xffffff00019feba0,
newtd=0xffffff0001883000, flags=Variable "flags" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850
46 Thread 100033 (PID=14: usb/usbus0) sched_switch (td=0xffffff0001a2a000,
newtd=0xffffff00019feba0, flags=Variable "flags" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850
45 Thread 100032 (PID=14: usb/usbus0) sched_switch (td=0xffffff0001a2a3e0,
newtd=0xffffff0001a2a000, flags=Variable "flags" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850
(kgdb) thread 121
[Switching to thread 121 (Thread 100418)]#0 sched_switch (
td=0xffffff00758633e0, newtd=0xffffff0005b40000, flags=Variable "flags" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850
1850 cpuid = PCPU_GET(cpuid);
(kgdb) bt
#0 sched_switch (td=0xffffff00758633e0, newtd=0xffffff0005b40000, flags=Variable "flags" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850
#1 0xffffffff805f90ef in mi_switch (flags=259, newtd=0x0)
at /data2v/home/nox/src-r81/src/sys/kern/kern_synch.c:449
#2 0xffffffff80630fb6 in turnstile_wait (ts=Variable "ts" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/subr_turnstile.c:746
#3 0xffffffff805e11c0 in _mtx_lock_sleep (m=0xffffff8000a6c330,
tid=18446742976169653216, opts=Variable "opts" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/kern_mutex.c:447
#4 0xffffffff805e14b3 in _mtx_lock_flags (m=Variable "m" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/kern_mutex.c:203
#5 0xffffffff8117839b in run_node_cleanup (ni=0xffffff8000f83000)
at /data2v/home/nox/src-r81/src/sys/modules/usb/run/../../../dev/usb/wlan/if_run.c:1719
#6 0xffffffff806db816 in ieee80211_sta_leave (ni=0xffffff8000f83000)
at /data2v/home/nox/src-r81/src/sys/net80211/ieee80211_node.c:834
#7 0xffffffff806db94e in ieee80211_node_leave (ni=0xffffff8000f83000)
---Type <return> to continue, or q <return> to quit---
at /data2v/home/nox/src-r81/src/sys/net80211/ieee80211_node.c:2508
#8 0xffffffff806d2c13 in setmlme_common (vap=0xffffff013e1e2000, op=Variable "op" is not available.
)
at /data2v/home/nox/src-r81/src/sys/net80211/ieee80211_ioctl.c:1327
#9 0xffffffff806d2db5 in ieee80211_ioctl_setmlme (vap=0xffffff013e1e2000,
ireq=Variable "ireq" is not available.
) at /data2v/home/nox/src-r81/src/sys/net80211/ieee80211_ioctl.c:1512
#10 0xffffffff806d405a in ieee80211_ioctl_set80211 (vap=0xffffff013e1e2000,
cmd=Variable "cmd" is not available.
) at /data2v/home/nox/src-r81/src/sys/net80211/ieee80211_ioctl.c:2721
#11 0xffffffff806f7b7b in in_control (so=0xffffff01e7ef3d48, cmd=2149607914,
data=0xffffff0007832460 "wlan0", ifp=0xffffff013e2c7800,
td=0xffffff00758633e0)
at /data2v/home/nox/src-r81/src/sys/netinet/in.c:290
#12 0xffffffff806a27b7 in ifioctl (so=0xffffff01e7ef3d48, cmd=2149607914,
data=0xffffff0007832460 "wlan0", td=0xffffff00758633e0)
at /data2v/home/nox/src-r81/src/sys/net/if.c:2523
#13 0xffffffff80632bc6 in kern_ioctl (td=0xffffff00758633e0, fd=3,
com=2149607914, data=0xffffff0007832460 "wlan0") at file.h:262
#14 0xffffffff80632e0d in ioctl (td=0xffffff00758633e0,
---Type <return> to continue, or q <return> to quit---
uap=0xffffff80ee69ebf0)
at /data2v/home/nox/src-r81/src/sys/kern/sys_generic.c:678
#15 0xffffffff808e5407 in syscall (frame=0xffffff80ee69ec80)
at /data2v/home/nox/src-r81/src/sys/amd64/amd64/trap.c:945
#16 0xffffffff808cac31 in Xfast_syscall ()
at /data2v/home/nox/src-r81/src/sys/amd64/amd64/exception.S:374
#17 0x0000000800ca438c in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) fr 5
#5 0xffffffff8117839b in run_node_cleanup (ni=0xffffff8000f83000)
at /data2v/home/nox/src-r81/src/sys/modules/usb/run/../../../dev/usb/wlan/if_run.c:1719
1719 RUN_LOCK(sc);
(kgdb) l
1714 wcid = rn->wcid;
1715 /* sc_ni[0] is not used */
1716 if (wcid != 0 && wcid <= RT2870_WCID_MAX)
1717 sc->sc_ni[wcid] = NULL;
1718 } else {
1719 RUN_LOCK(sc);
1720 wcid = rn->wcid;
1721 if (wcid != 0 && wcid <= RT2870_WCID_MAX)
1722 sc->sc_ni[wcid] = NULL;
1723 RUN_UNLOCK(sc);
(kgdb) down
#4 0xffffffff805e14b3 in _mtx_lock_flags (m=Variable "m" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/kern_mutex.c:203
203 _get_sleep_lock(m, curthread, opts, file, line);
(kgdb)
#3 0xffffffff805e11c0 in _mtx_lock_sleep (m=0xffffff8000a6c330,
tid=18446742976169653216, opts=Variable "opts" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/kern_mutex.c:447
447 turnstile_wait(ts, mtx_owner(m), TS_EXCLUSIVE_QUEUE);
(kgdb) p m
$1 = (struct mtx *) 0xffffff8000a6c330
(kgdb) p *m
$2 = {lock_object = {lo_name = 0xffffff0005e799e0 "run0",
lo_flags = 16973824, lo_data = 0, lo_witness = 0x0},
mtx_lock = 18446742974292827042}
(kgdb) p m.mtx_lock & 0xfffffffffffffff
$3 = 1152920405190122402
(kgdb) p m.mtx_lock & 0xffffffffffffffff
$4 = 18446742974292827042
(kgdb) p m.mtx_lock & 0xfffffffffffffff8
$5 = 18446742974292827040
(kgdb) p (struct thread *)m.mtx_lock & 0xfffffffffffffff8
Argument to arithmetic operation not a number or boolean.
(kgdb) p (struct thread *)(m.mtx_lock & 0xfffffffffffffff8)
$6 = (struct thread *) 0xffffff0005a81ba0
(kgdb) thr 54
[Switching to thread 54 (Thread 100041)]#0 sched_switch (
td=0xffffff0005a81ba0, newtd=0xffffff0001883000, flags=Variable "flags" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850
1850 cpuid = PCPU_GET(cpuid);
(kgdb) bt
#0 sched_switch (td=0xffffff0005a81ba0, newtd=0xffffff0001883000, flags=Variable "flags" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850
#1 0xffffffff805f90ef in mi_switch (flags=259, newtd=0x0)
at /data2v/home/nox/src-r81/src/sys/kern/kern_synch.c:449
#2 0xffffffff80630fb6 in turnstile_wait (ts=Variable "ts" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/subr_turnstile.c:746
#3 0xffffffff805e11c0 in _mtx_lock_sleep (m=0xffffff8000a717c8,
tid=18446742974292827040, opts=Variable "opts" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/kern_mutex.c:447
#4 0xffffffff806dad00 in ieee80211_free_node (ni=0xffffff8000f83000)
at /data2v/home/nox/src-r81/src/sys/net80211/ieee80211_node.c:1682
#5 0xffffffff81172e1a in run_tx_free (pq=0xffffff8000a6c350,
data=0xffffff8000a6c660, txerr=Variable "txerr" is not available.
)
at /data2v/home/nox/src-r81/src/sys/modules/usb/run/../../../dev/usb/wlan/if_run.c:2759
#6 0xffffffff8117783d in run_bulk_tx_callbackN (xfer=0xffffff8000d1e148,
error=USB_ERR_NORMAL_COMPLETION, index=0)
---Type <return> to continue, or q <return> to quit---
at /data2v/home/nox/src-r81/src/sys/modules/usb/run/../../../dev/usb/wlan/if_run.c:2793
#7 0xffffffff8052a92d in usbd_callback_wrapper (pq=Variable "pq" is not available.
)
at /data2v/home/nox/src-r81/src/sys/dev/usb/usb_transfer.c:2136
#8 0xffffffff80526fa6 in usb_command_wrapper (pq=0xffffff8000d1e060, xfer=Variable "xfer" is not available.
)
at /data2v/home/nox/src-r81/src/sys/dev/usb/usb_transfer.c:2745
#9 0xffffffff80529a70 in usb_callback_proc (_pm=Variable "_pm" is not available.
)
at /data2v/home/nox/src-r81/src/sys/dev/usb/usb_transfer.c:2005
#10 0xffffffff80524633 in usb_process (arg=Variable "arg" is not available.
)
at /data2v/home/nox/src-r81/src/sys/dev/usb/usb_process.c:166
#11 0xffffffff805c64a8 in fork_exit (
callout=0xffffffff80524560 <usb_process>, arg=0xffffff80003e8d10,
frame=0xffffff80e97efc80)
at /data2v/home/nox/src-r81/src/sys/kern/kern_fork.c:844
#12 0xffffffff808cae2e in fork_trampoline ()
at /data2v/home/nox/src-r81/src/sys/amd64/amd64/exception.S:562
#13 0x0000000000000000 in ?? ()
---Type <return> to continue, or q <return> to quit---
#14 0x0000000000000000 in ?? ()
#15 0x0000000000000001 in ?? ()
#16 0x0000000000000000 in ?? ()
#17 0x0000000000000000 in ?? ()
#18 0x0000000000000000 in ?? ()
#19 0x0000000000000000 in ?? ()
#20 0x0000000000000000 in ?? ()
#21 0x0000000000000000 in ?? ()
#22 0x0000000000000000 in ?? ()
#23 0x0000000000000000 in ?? ()
#24 0x0000000000000000 in ?? ()
#25 0x0000000000000000 in ?? ()
#26 0x0000000000000000 in ?? ()
#27 0x0000000000000000 in ?? ()
#28 0x0000000000000000 in ?? ()
#29 0x0000000000000000 in ?? ()
#30 0x0000000000000000 in ?? ()
---Type <return> to continue, or q <return> to quit---q
Quit
(kgdb) fr 3
#3 0xffffffff805e11c0 in _mtx_lock_sleep (m=0xffffff8000a717c8,
tid=18446742974292827040, opts=Variable "opts" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/kern_mutex.c:447
447 turnstile_wait(ts, mtx_owner(m), TS_EXCLUSIVE_QUEUE);
(kgdb) p m
$7 = (struct mtx *) 0xffffff8000a717c8
(kgdb) p *m
$8 = {lock_object = {lo_name = 0xffffff8000a717b8 "run0_node_lock",
lo_flags = 17498112, lo_data = 0, lo_witness = 0x0},
mtx_lock = 18446742976169653218}
(kgdb) p (struct thread *)(m.mtx_lock & 0xfffffffffffffff8)
$9 = (struct thread *) 0xffffff00758633e0
(kgdb) thread 121
[Switching to thread 121 (Thread 100418)]#0 sched_switch (
td=0xffffff00758633e0, newtd=0xffffff0005b40000, flags=Variable "flags" is not available.
)
at /data2v/home/nox/src-r81/src/sys/kern/sched_ule.c:1850
1850 cpuid = PCPU_GET(cpuid);
(kgdb) q
More information about the freebsd-net
mailing list