7.1-STABLE crash
Asmodean Dark
asmodean at list.ru
Tue Jun 2 09:41:45 UTC 2009
I`ve attached the requested information. I hope this helps.
-----Original Message-----
From: Mikolaj Golub <to.my.trociny at gmail.com>
To: Asmodean Dark <asmodean at list.ru>
Date: Tue, 02 Jun 2009 10:46:23 +0300
Subject: Re: 7.1-STABLE crash
On Tue, 02 Jun 2009 10:36:32 +0400 Asmodean Dark wrote:
>>> Do you really need a such outdated version of mpd while mpd5 is available in ports?
> Yes, we need mpd3 because there are many our custom patches for it (some RADIUS functions, bugfixes etc).
>
>>> Do you have some automatic blocker or some other script that periodically add/remove IPs in a ipfw table?
> Yes. For some our users we use ipfw-based rules (SMTP port blocking, enabling transparent proxy and other). Mpd ifaceup script contain something like this:
> if [ "$FilterId" = "no_transparent" ]; then
> $IPFW table 3 add $IP
> fi
>
> And, in ipfw.rules:
> add allow tcp from table(3) to any dst-port 80 in recv ng*
>
> Why table can be corrupted? ipfw check added address, isn`t it?
It might be that when ipfw was looking for IP in the table it was being
modified by ifaceup script at that time. ipfw has lock protection so this
thing should not have happened but...
Could you run something like this?
ps -auxl -M /path/to/vmcore -N /path/to/kernel.symbols
We could look if at the moment of the crash some process was running that was
adding/removing ipfw table.
Other good and simple thing is to run crashinfo(8) utility and provide its
output. If it runs flawlessly it should contain ps output too among other
useful information.
>>> Do the bt output for other crashes looks the same?
> It`s a first dump obtained. There is some problems in receiving it, because all servers is network-booted (via PXE) and have no usable dump devices This dump obtained with USB flash device connected to server. Also, crashes is not often.
So we could try to get as much info from this dump as we can . Could you
post here the whole output of kgdb from its start, not only bt (I mean things
like "Reading symbols from..." and the command line itself)?
Also you can try in kgdb session:
fr 9
list
p *cmd
p cmd->arg1
p a
-------------- next part --------------
A non-text attachment was scrubbed...
Name: crashinfo.tar.bz2
Type: application/octet-stream
Size: 99980 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20090602/31e309b5/crashinfo.tar-0001.obj
-------------- next part --------------
# LC_ALL=C ps -auxl -M /tmp/crash/vmcore.0 -N /tmp/crash/kernel.symbols
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND UID PPID CPU PRI NI MWCHAN
root 0 0.0 0.0 0 0 ?? DLs 1Jan70 -28302384:-40.55 [swapper] 0 0 0 -16 0 sched
root 1 0.0 0.0 1888 532 ?? DLs 1Jan70 -22953083:-22.55 [init] 0 0 0 8 0 wait
root 2 0.0 0.0 0 0 ?? DL 1Jan70 -35018924:-59.55 [g_event] 0 0 0 -8 0 -
root 3 0.0 0.0 0 0 ?? DL 1Jan70 19733865:36.00 [g_up] 0 0 0 -8 0 -
root 4 0.0 0.0 0 0 ?? DL 1Jan70 -21084496:00.55 [g_down] 0 0 0 -8 0 -
root 5 0.0 0.0 0 0 ?? DL 1Jan70 0:00.00 [xpt_thrd] 0 0 0 -8 0 ccb_sc
root 6 0.0 0.0 0 0 ?? DL 1Jan70 0:00.00 [acpi_task_0] 0 0 0 8 0 -
root 7 0.0 0.0 0 0 ?? DL 1Jan70 0:00.00 [acpi_task_1] 0 0 0 8 0 -
root 8 0.0 0.0 0 0 ?? DL 1Jan70 0:00.00 [acpi_task_2] 0 0 0 8 0 -
root 9 0.0 0.0 0 0 ?? DL 1Jan70 0:00.00 [kqueue taskq] 0 0 0 8 0 -
root 10 0.0 0.0 0 0 ?? RL 1Jan70 22749128:30.00 [idle: cpu1] 0 0 0 171 0 -
root 11 0.0 0.0 0 0 ?? RL 1Jan70 9219592:17.00 [idle: cpu0] 0 0 0 171 0 -
root 12 0.0 0.0 0 0 ?? RL 1Jan70 14087762:45.00 [swi4: clock] 0 0 0 -32 0 -
root 13 0.0 0.0 0 0 ?? WL 1Jan70 0:00.00 [swi3: vm] 0 0 0 -36 0 -
root 14 0.0 0.0 0 0 ?? RL 1Jan70 17900236:59.00 [swi1: net] 0 0 0 -44 0 -
root 15 0.0 0.0 0 0 ?? DL 1Jan70 -29370189:00.55 [yarrow] 0 0 0 44 0 -
root 16 0.0 0.0 0 0 ?? WL 1Jan70 0:00.00 [swi6: task queu 0 0 0 -24 0 -
root 17 0.0 0.0 0 0 ?? WL 1Jan70 274009:24.00 [swi2: cambio] 0 0 0 -40 0 -
root 18 0.0 0.0 0 0 ?? WL 1Jan70 0:00.00 [swi5: +] 0 0 0 -28 0 -
root 19 0.0 0.0 0 0 ?? DL 1Jan70 188:53.00 [thread taskq] 0 0 0 8 0 -
root 20 0.0 0.0 0 0 ?? WL 1Jan70 4600168:46.00 [swi6: Giant tas 0 0 0 -24 0 -
root 21 0.0 0.0 0 0 ?? WL 1Jan70 0:00.00 [irq9: acpi0] 0 0 0 -52 0 -
root 22 0.0 0.0 0 0 ?? RL 1Jan70 -10575708:-22.55 [em0 taskq] 0 0 0 -68 0 -
root 23 0.0 0.0 0 0 ?? WL 1Jan70 0:00.00 [irq18: em1 uhci 0 0 0 -68 0 -
root 24 0.0 0.0 0 0 ?? DL 1Jan70 3264118:40.00 [usb0] 0 0 0 8 0 usbevt
root 25 0.0 0.0 0 0 ?? DL 1Jan70 0:00.00 [usbtask-hc] 0 0 0 8 0 usbtsk
root 26 0.0 0.0 0 0 ?? DL 1Jan70 0:00.00 [usbtask-dr] 0 0 0 8 0 usbtsk
root 27 0.0 0.0 0 0 ?? WL 1Jan70 0:00.00 [irq21: uhci1+] 0 0 0 -64 0 -
root 28 0.0 0.0 0 0 ?? DL 1Jan70 3607326:48.00 [usb1] 0 0 0 8 0 usbevt
root 29 0.0 0.0 0 0 ?? WL 1Jan70 0:00.00 [irq17: uhci2 eh 0 0 0 -64 0 -
root 30 0.0 0.0 0 0 ?? DL 1Jan70 3381869:17.00 [usb2] 0 0 0 8 0 usbevt
root 31 0.0 0.0 0 0 ?? DL 1Jan70 3518333:14.00 [usb3] 0 0 0 8 0 usbevt
root 32 0.0 0.0 0 0 ?? RL 1Jan70 672675:23.00 [irq23: uhci3 eh 0 0 0 -64 0 -
root 33 0.0 0.0 0 0 ?? DL 1Jan70 3256232:07.00 [usb4] 0 0 0 8 0 usbevt
root 34 0.0 0.0 0 0 ?? WL 1Jan70 0:00.00 [irq19: uhci4] 0 0 0 -64 0 -
root 35 0.0 0.0 0 0 ?? DL 1Jan70 3301012:03.00 [usb5] 0 0 0 8 0 usbevt
root 36 0.0 0.0 0 0 ?? DL 1Jan70 3376947:56.00 [usb6] 0 0 0 8 0 usbevt
root 37 0.0 0.0 0 0 ?? DL 1Jan70 4143595:52.00 [usb7] 0 0 0 8 0 usbevt
root 38 0.0 0.0 0 0 ?? DL 1Jan70 0:00.00 [em1 taskq] 0 0 0 -68 0 -
root 39 0.0 0.0 0 0 ?? WL 1Jan70 17026944:14.00 [irq14: ata0] 0 0 0 -64 0 -
root 40 0.0 0.0 0 0 ?? WL 1Jan70 0:00.00 [irq15: ata1] 0 0 0 -64 0 -
root 41 0.0 0.0 0 0 ?? WL 1Jan70 453060:39.00 [irq1: atkbd0] 0 0 0 -60 0 -
root 42 0.0 0.0 0 0 ?? DL 1Jan70 2888630:00.00 [dummynet] 0 0 0 -68 0 -
root 43 0.0 0.0 0 0 ?? DL 1Jan70 537454:24.00 [md0] 0 0 0 -8 0 mdwait
root 44 0.0 0.0 0 0 ?? DL 1Jan70 -21141059:-47.55 [pagedaemon] 0 0 0 -16 0 psleep
root 45 0.0 0.0 0 0 ?? DL 1Jan70 75:57.00 [vmdaemon] 0 0 0 20 0 psleep
root 46 0.0 0.0 0 0 ?? DL 1Jan70 737245:15.00 [pagezero] 0 0 0 171 0 pgzero
root 47 0.0 0.0 0 0 ?? DL 1Jan70 -5215608:-59.55 [idlepoll] 0 0 0 171 0 pollid
root 48 0.0 0.0 0 0 ?? DL 1Jan70 -35220794:-43.55 [bufdaemon] 0 0 0 -16 0 psleep
root 49 0.0 0.0 0 0 ?? DL 1Jan70 3263896:16.00 [vnlru] 0 0 0 -4 0 vlruwt
root 50 0.0 0.0 0 0 ?? DL 1Jan70 -14008286:-51.55 [syncer] 0 0 0 20 0 syncer
root 51 0.0 0.0 0 0 ?? DL 1Jan70 -7574436:-19.55 [softdepflush] 0 0 0 -16 0 sdflus
root 118 0.0 0.0 0 0 ?? DL 1Jan70 -26345438:-14.55 [md1] 0 0 0 -8 0 mdwait
root 280 0.0 0.0 3088 64 ?? D 1Jan70 5188057:38.00 [dhclient] 0 1 0 48 0 select
_dhcp 302 0.0 0.0 3088 64 ?? Ds 1Jan70 27402661:43.00 [dhclient] 65 1 0 44 0 select
root 396 0.0 0.0 0 0 ?? DL 1Jan70 -30322983:-22.55 [md5] 0 0 0 -8 0 mdwait
root 404 0.0 0.0 0 0 ?? DL 1Jan70 28645466:01.00 [md10] 0 0 0 -8 0 mdwait
root 412 0.0 0.0 0 0 ?? DL 1Jan70 -25819269:-55.55 [md15] 0 0 0 -8 0 mdwait
root 420 0.0 0.0 0 0 ?? DL 1Jan70 273691:15.00 [md20] 0 0 0 -8 0 mdwait
root 510 0.0 0.0 0 0 ?? DL 1Jan70 992311:50.00 [md21] 0 0 0 -8 0 mdwait
root 554 0.0 0.0 3300 32 ?? Ds 1Jan70 -17645368:-45.55 [rpcbind] 0 1 0 44 0 select
root 556 0.0 0.0 266288 16 ?? Ds 1Jan70 -26070283:-44.55 [rpc.statd] 0 1 0 44 0 select
root 569 0.0 0.0 6248 44 ?? Ds 1Jan70 -25412810:-36.55 [rpc.lockd] 0 1 0 44 0 select
daemon 573 0.0 0.0 6248 44 ?? D 1Jan70 9510:40.00 [rpc.lockd] 1 569 0 4 0 nfsloc
root 946 0.0 0.0 3144 32 ?? Rs 1Jan70 27673326:45.00 [syslogd] 0 1 0 44 0 -
quagga 958 0.0 0.0 4812 152 ?? Ds 1Jan70 -7630094:-54.55 [zebra] 101 1 0 44 0 select
quagga 964 0.0 0.0 15176 8 ?? Rs 1Jan70 8082176:22.00 [ospfd] 101 1 0 44 0 -
root 1062 0.0 0.0 7000 4 ?? D 1Jan70 -28023279:-25.55 [perl] 0 1 0 4 0 accept
root 1084 0.0 0.0 65992 232 ?? Rs 1Jan70 -19135518:-47.55 [mpd] 0 1 0 62 0 -
root 1094 0.0 0.0 44728 24 ?? D 1Jan70 31368572:49.00 [snmpd] 0 1 0 44 0 select
nagios 1100 0.0 0.0 3088 20 ?? Ds 1Jan70 30057867:43.00 [nrpe] 1023 1 0 4 0 accept
root 1140 0.0 0.0 5776 392 ?? Ds 1Jan70 -9652890:-8.55 [ntpd] 0 1 0 44 0 select
root 1294 0.0 0.0 4948 188 ?? Ds 1Jan70 34498147:01.55 [sshd] 0 1 0 44 0 select
root 1308 0.0 0.0 3172 32 ?? Ds 1Jan70 31886589:41.00 [cron] 0 1 0 8 0 nanslp
root 1333 0.0 0.0 3420 108 ?? D 1Jan70 33952404:01.55 [sh] 0 1 0 8 0 wait
root 1401 0.0 0.0 3144 20 ?? Ds+ 1Jan70 45271:13.00 [getty] 0 1 0 5 0 ttyin
root 1402 0.0 0.0 3144 20 ?? Ds+ 1Jan70 40783:24.00 [getty] 0 1 0 5 0 ttyin
root 1403 0.0 0.0 3144 20 ?? Ds+ 1Jan70 43523:33.00 [getty] 0 1 0 5 0 ttyin
root 1404 0.0 0.0 3144 20 ?? Ds+ 1Jan70 39453:31.00 [getty] 0 1 0 5 0 ttyin
root 1405 0.0 0.0 3144 20 ?? Ds+ 1Jan70 40387:05.00 [getty] 0 1 0 5 0 ttyin
root 1406 0.0 0.0 3144 20 ?? Ds+ 1Jan70 39278:45.00 [getty] 0 1 0 5 0 ttyin
root 1407 0.0 0.0 3144 20 ?? Ds+ 1Jan70 39826:16.00 [getty] 0 1 0 5 0 ttyin
root 1408 0.0 0.0 3144 20 ?? Ds+ 1Jan70 40036:44.00 [getty] 0 1 0 5 0 ttyin
root 69523 0.0 0.0 1336 4 ?? D 1Jan70 0:00.00 [sleep] 0 1333 0 8 0 nanslp
-------------- next part --------------
# kgdb kernel.debug vmcore.0
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...No struct type named linker_file.
No struct type named linker_file.
No struct type named linker_file.
Attempt to extract a component of a value that is not a structure.
No struct type named linker_file.
No struct type named linker_file.
No struct type named linker_file.
Attempt to extract a component of a value that is not a structure.
Attempt to extract a component of a value that is not a structure pointer.
Attempt to extract a component of a value that is not a structure pointer.
Attempt to extract a component of a value that is not a structure pointer.
Attempt to extract a component of a value that is not a structure pointer.
#0 0x8063d6b0 in doadump ()
(kgdb) bt
#0 0x8063d6b0 in doadump ()
#1 0x8063dc44 in boot ()
#2 0x8063e0ca in panic ()
#3 0x807dab3d in trap_fatal ()
#4 0x807daeba in trap_pfault ()
#5 0x807db7bd in trap ()
#6 0x807c2a3b in calltrap ()
#7 0x806dcb88 in rn_match ()
#8 0x806ddc8a in rn_lookup ()
#9 0x8070e460 in ipfw_chk (args=0xe70175fc) at ../../../netinet/ip_fw2.c:1894
#10 0x80710c3d in ipfw_check_in (arg=0x0, m0=0xe7017700, ifp=0x91c5a800, dir=1, inp=0x0) at ../../../netinet/ip_fw_pfil.c:125
#11 0x806dc20f in pfil_run_hooks ()
#12 0x80713984 in ip_input (m=0x91954c00) at ../../../netinet/ip_input.c:416
#13 0x806ec0d9 in ng_iface_rcvdata ()
#14 0x806e9570 in ng_apply_item ()
#15 0x806e8569 in ng_snd_item ()
#16 0x806e9570 in ng_apply_item ()
#17 0x806e8569 in ng_snd_item ()
#18 0x806e9570 in ng_apply_item ()
#19 0x806e8569 in ng_snd_item ()
#20 0x806f16a7 in ng_ppp_proto_recv ()
#21 0x806f3ed2 in ng_ppp_rcvdata ()
#22 0x806e9570 in ng_apply_item ()
#23 0x806e8569 in ng_snd_item ()
#24 0x806e9570 in ng_apply_item ()
#25 0x806e8569 in ng_snd_item ()
#26 0x806ee3c3 in ng_ksocket_incoming2 ()
#27 0x806e969d in ng_apply_item ()
#28 0x806ea8aa in ngintr ()
#29 0x806dab72 in swi_net ()
#30 0x8061e265 in ithread_loop ()
#31 0x8061adf5 in fork_exit ()
#32 0x807c2ab0 in fork_trampoline ()
(kgdb) fr 9
#9 0x8070e460 in ipfw_chk (args=0xe70175fc) at ../../../netinet/ip_fw2.c:1894
1894 sa.sin_len = 8;
(kgdb) list
1889 struct sockaddr_in sa;
1890
1891 if (tbl >= IPFW_TABLES_MAX)
1892 return (0);
1893 rnh = ch->tables[tbl];
1894 sa.sin_len = 8;
1895 sa.sin_addr.s_addr = addr;
1896 ent = (struct table_entry *)(rnh->rnh_lookup(&sa, NULL, rnh));
1897 if (ent != NULL) {
1898 *val = ent->value;
(kgdb) p *cmd
$1 = {opcode = O_IP_SRC_LOOKUP, len = 1 '\001', arg1 = 2}
(kgdb) p cmd->arg1
$2 = 2
(kgdb) p a
Variable "a" is not available.
More information about the freebsd-hackers
mailing list