misc/185092: panic: rtfree 2 (using RADIX_MPATH in a VNET jail)
Nikolay Denev
ndenev at gmail.com
Wed Jan 1 17:39:46 UTC 2014
Ok, killing openvpn with -9 leaves the routes around, and particularly
interesting are the following routes :
78.90.222.xx 10.255.255.0 UGHS 0 5841 epair0 =>
78.90.222.xx/32 10.255.255.0 UGS 0 0 epair0
Now, if I do :
route delete 78.90.222.xx 10.255.255.0
The route, with the H flag is deleted. If I repeat the command the
second route is deleted as well, even if the second command specifies
a netmask no panic.
However the first delete command specifies the /32 mask like this :
route delete 78.90.222.xx 10.255.255.0 255.255.255.255
Then I get "rtfree 2" kernel panic immediately.
This seems to be happening as I'm manually installing static routes in
the vnet jail for the VPN remote endpoints , however OpenVPN adds such
routes too however differently, which results in two routing entries.
For example :
route add $IP $GW
and
route add $IP $GW 255.255.255.255
add to different route entries, one is /32 network, the other is a host route.
--Nikolay
On Wed, Jan 1, 2014 at 1:21 PM, Nikolay Denev <ndenev at gmail.com> wrote:
> On Wed, Jan 1, 2014 at 1:10 PM, Nikolay Denev <ndenev at gmail.com> wrote:
>>
>> On Sun, Dec 22, 2013 at 1:10 PM, <FreeBSD-gnats-submit at freebsd.org> wrote:
>>>
>>> Thank you very much for your problem report.
>>> It has the internal identification `misc/185092'.
>>> The individual assigned to look at your
>>> report is: freebsd-bugs.
>>>
>>> You can access the state of your problem report at any time
>>> via this link:
>>>
>>> http://www.freebsd.org/cgi/query-pr.cgi?pr=185092
>>>
>>> >Category: misc
>>> >Responsible: freebsd-bugs
>>> >Synopsis: panic: rtfree 2 (using RADIX_MPATH in a VNET jail)
>>> >Arrival-Date: Sun Dec 22 13:10:00 UTC 2013
>>
>>
>> I'm trying to understand exactly what is happening here, and examining a
>> core dump with kgdb I'm getting some output that confuses me :
>>
>> (kgdb) bt
>> #0 doadump (textdump=-1011569920) at pcpu.h:233
>> #1 0xc06069b2 in kern_reboot (howto=260) at
>> /usr/src/sys/kern/kern_shutdown.c:447
>> #2 0xc0606d0e in panic (fmt=<value optimized out>) at
>> /usr/src/sys/kern/kern_shutdown.c:754
>> #3 0xc06de639 in rtfree (rt=<value optimized out>) at
>> /usr/src/sys/net/route.c:464
>> #4 0xc06e188d in route_output (m=<value optimized out>) at
>> /usr/src/sys/net/rtsock.c:951
>> #5 0xc06de18f in raw_usend (so=<value optimized out>, flags=0, m=<value
>> optimized out>, nam=0x0, control=<value optimized out>,
>> td=0xc3bd2000) at /usr/src/sys/net/raw_usrreq.c:238
>> #6 0xc066eca9 in sosend_generic (so=0xc3e9c1a8, uio=<value optimized
>> out>, top=<value optimized out>, control=0x0,
>> flags=<value optimized out>, td=<value optimized out>) at
>> /usr/src/sys/kern/uipc_socket.c:1271
>> #7 0xc066efc7 in sosend (so=0xc3e9c1a8, addr=0x0, uio=0xd9b9cc10,
>> top=0x0, control=0x0, flags=0, td=0xc3bd2000)
>> at /usr/src/sys/kern/uipc_socket.c:1315
>> #8 0xc0654af4 in soo_write (fp=0xc3c0c818, uio=0xd9b9cc10,
>> active_cred=0xc3f1dd00, flags=0, td=0xc3bd2000)
>> at /usr/src/sys/kern/sys_socket.c:103
>> #9 0xc064c866 in dofilewrite (td=0xc3bd2000, fd=3, fp=0xc3c0c818,
>> auio=0xd9b9cc10, offset=-1, flags=0) at file.h:303
>> #10 0xc064c566 in kern_writev (td=0xc3bd2000, fd=3, auio=<value optimized
>> out>) at /usr/src/sys/kern/sys_generic.c:467
>> #11 0xc064c4bc in sys_write (td=<value optimized out>, uap=<value
>> optimized out>) at /usr/src/sys/kern/sys_generic.c:382
>> #12 0xc08614d3 in syscall (frame=<value optimized out>) at
>> subr_syscall.c:134
>> #13 0xc084cca1 in Xint0x80_syscall () at
>> /usr/src/sys/i386/i386/exception.s:270
>> #14 0x281975b7 in ?? ()
>> Previous frame inner to this frame (corrupt stack?)
>> Current language: auto; currently minimal
>> (kgdb) fr 3
>> #3 0xc06de639 in rtfree (rt=<value optimized out>) at
>> /usr/src/sys/net/route.c:464
>> 464 panic("rtfree 2");
>> (kgdb) print *rt
>> $1 = {rt_nodes = {{rn_mklist = 0xc3b4ab30, rn_parent = 0x1, rn_bit = 0,
>> rn_bmask = 0 '\0', rn_flags = 0 '\0', rn_u = {rn_leaf = {
>> rn_Key = 0xc0882687 "shutdown_post_sync", rn_Mask = 0x1030000
>> <Address 0x1030000 out of bounds>, rn_Dupedkey = 0x0}, rn_node = {
>> rn_Off = -1064819065, rn_L = 0x1030000, rn_R = 0x0}}},
>> {rn_mklist = 0x0, rn_parent = 0x4, rn_bit = -18048, rn_bmask = -94 '?',
>> rn_flags = 195 '?', rn_u = {rn_leaf = {rn_Key = 0xc3a545e0 "",
>> rn_Mask = 0xc3a4e440 " ??(???\020'", rn_Dupedkey = 0xc3a4e880},
>> rn_node = {rn_Off = -1012578848, rn_L = 0xc3a4e440, rn_R =
>> 0xc3a4e880}}}}, rt_gateway = 0x74756873, rt_flags = 1853321060,
>> rt_refcnt = 1936683103, rt_ifp = 0x79735f74, rt_ifa = 0x636e, rt_rmx =
>> {rmx_mtu = 0, rmx_expire = 0, rmx_pksent = 0, rmx_weight = 0},
>> rt_fibnum = 0, rt_mtx = {lock_object = {lo_name = 0x0, lo_flags = 0,
>> lo_data = 0, lo_witness = 0x0}, mtx_lock = 0}}
>>
>>
>>
>> rn_Key with value of “shutdown_post_sync” ?
>>
>> It’s visible also in the raw_usend() frame:
>>
>> (kgdb) fr 5
>> #5 0xc06de18f in raw_usend (so=<value optimized out>, flags=0, m=<value
>> optimized out>, nam=0x0, control=<value optimized out>,
>> td=0xc3bd2000) at /usr/src/sys/net/raw_usrreq.c:238
>> 238 return ((*so->so_proto->pr_output)(m, so));
>> (kgdb) print *m
>> $2 = {m_hdr = {mh_next = 0xc3b4ab30, mh_nextpkt = 0x1, mh_data = 0x0,
>> mh_len = -1064819065, mh_type = 0, mh_flags = 66304, mh_pad = 0},
>> M_dat = {MH = {MH_pkthdr = {rcvif = 0x0, tags = {slh_first = 0x4}, len =
>> -1012745856, flowid = 3282388448,
>> csum_flags = 14097648373312316480, fibnum = 26739, cosqos = 117
>> 'u', rsstype = 116 't', l2hlen = 100 'd', l3hlen = 111 'o',
>> l4hlen = 119 'w', l5hlen = 110 'n', PH_per = {eigth = "_post_sy",
>> sixteen = {28767, 29551, 24436, 31091}, thirtytwo = {1936683103,
>> 2037604212}, sixtyfour = {8751443454668533855}, unintptr =
>> {1936683103}, ptr = 0x736f705f}, PH_loc = {
>> eigth = "nc\000\000\000\000\000", sixteen = {25454, 0, 0, 0},
>> thirtytwo = {25454, 0}, sixtyfour = {25454}, unintptr = {25454},
>> ptr = 0x636e}}, MH_dat = {MH_ext = {ref_cnt = 0x0, ext_buf =
>> 0x0, ext_size = 0, ext_type = 0, ext_flags = 0, ext_free = 0,
>> ext_arg1 = 0x0, ext_arg2 = 0x0},
>> MH_databuf = '\0' <repeats 56 times>, "file", '\0' <repeats 20
>> times>,
>> "\006\000\000\000\020\000\000\000??\215?\000\000C\001\000\000\000\000\000\000\000\000\004\000\000\000\000\000\000\00000Y?",
>> '\0' <repeats 12 times>,
>> "`2Y?\000\000\000\000\000\000\000\000T\211\223?\022\000\000\000\000\203??\000\000\000\000\000???",
>> '\0' <repeats 23 times>}},
>> M_databuf =
>> "\000\000\000\000\004\000\000\000\200????E??@??\200??shutdown_post_sync",
>> '\0' <repeats 62 times>, "file", '\0' <repeats 20 times>,
>> "\006\000\000\000\020\000\000\000??\215?\000\000C\001\000\000\000\000\000\000\000\000\004\000\000\000\000\000\000\00000Y?",
>> '\0' <repeats 12 times>,
>> "`2Y?\000\000\000\000\000\000\000\000T\211\223?\022\000\000\000\000\203??\000\000\000\000\000???",
>> '\0' <repeats 23 times>}}
>>
>>
>> This is 10.0-PRERELEASE r259547M (with applied the recent nd6_nbr.c rtfree
>> patch, which I thought earlier might be the cause of the panics I'm
>> seeing).
>>
>> The machine is Soekris Net5501-70 with this kernel config :
>>
>> cpu I586_CPU
>> cpu I686_CPU
>> ident MARS
>> options CPU_GEODE
>> options CPU_SOEKRIS
>>
>> options HZ=2000
>> options DEVICE_POLLING
>> options BPF_JITTER
>>
>> makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols
>>
>> options SCHED_ULE # ULE scheduler
>> options PREEMPTION # Enable kernel thread preemption
>> options INET # InterNETworking
>> options INET6 # IPv6 communications protocols
>> options TCP_OFFLOAD # TCP offload
>> options FFS # Berkeley Fast Filesystem
>> options SOFTUPDATES # Enable FFS soft updates support
>> options UFS_DIRHASH # Improve performance on big directories
>> options PROCFS # Process filesystem (requires PSEUDOFS)
>> options PSEUDOFS # Pseudo-filesystem framework
>> options GEOM_PART_GPT # GUID Partition Tables.
>> options GEOM_LABEL # Provides labelization
>> options COMPAT_FREEBSD4 # Compatible with FreeBSD4
>> options COMPAT_FREEBSD5 # Compatible with FreeBSD5
>> options COMPAT_FREEBSD6 # Compatible with FreeBSD6
>> options COMPAT_FREEBSD7 # Compatible with FreeBSD7
>> options SCSI_DELAY=500 # Delay (in ms) before probing SCSI
>> options KTRACE # ktrace(1) support
>> options STACK # stack(9) support
>> options SYSVSHM # SYSV-style shared memory
>> options SYSVMSG # SYSV-style message queues
>> options SYSVSEM # SYSV-style semaphores
>> options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
>> options PRINTF_BUFR_SIZE=128 # Prevent printf output being interspersed.
>> options KBD_INSTALL_CDEV # install a CDEV entry in /dev
>> options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4)
>> options CAPABILITY_MODE # Capsicum capability mode
>> options CAPABILITIES # Capsicum capabilities
>> options PROCDESC # Support for process descriptors
>> options INCLUDE_CONFIG_FILE # Include this file in kernel
>>
>> # Debugging support. Always need this:
>> options KDB # Enable kernel debugger support.
>> options KDB_TRACE # Print a stack trace for a panic.
>> options KDB_UNATTENDED
>>
>> options TEXTDUMP_PREFERRED
>> options TEXTDUMP_VERBOSE
>>
>> device pci
>> device ata # Legacy ATA/SATA controllers
>> options ATA_STATIC_ID # Static device numbering
>>
>> # ATA/SCSI peripherals
>> device scbus # SCSI bus (required for ATA/SCSI)
>> device da # Direct Access (disks)
>> device pass # Passthrough device (direct ATA/SCSI access)
>>
>> # Add suspend/resume support for the i8254.
>> device pmtimer
>>
>> # Serial (COM) ports
>> device uart # Generic UART driver
>>
>> device miibus # MII bus support
>> device vr # VIA Rhine, Rhine II
>>
>> # Wireless NIC cards
>> device wlan # 802.11 support
>> options IEEE80211_DEBUG # enable debug msgs
>> options IEEE80211_AMPDU_AGE # age frames in AMPDU reorder q's
>> options IEEE80211_SUPPORT_MESH # enable 802.11s draft support
>> device wlan_wep # 802.11 WEP support
>> device wlan_ccmp # 802.11 CCMP support
>> device wlan_tkip # 802.11 TKIP support
>> device wlan_amrr # AMRR transmit rate control algorithm
>> device ath # Atheros NICs
>> device ath_pci # Atheros pci/cardbus glue
>> device ath_hal # pci/cardbus chip support
>> options AH_SUPPORT_AR5416 # enable AR5416 tx/rx descriptors
>> options AH_AR5416_INTERRUPT_MITIGATION # AR5416 interrupt mitigation
>> options ATH_ENABLE_11N # Enable 802.11n support for AR5416 and later
>> device ath_rate_sample # SampleRate tx rate control for ath
>>
>> # Pseudo devices.
>> device loop # Network loopback
>> device random # Entropy device
>> device ether # Ethernet support
>> device vlan # 802.1Q VLAN support
>> device tun # Packet tunnel.
>> device md # Memory "disks"
>> device gif # IPv6 and IPv4 tunneling
>> device gre
>> device faith # IPv6-to-IPv4 relaying (translation)
>> device firmware # firmware assist module
>> device if_bridge
>>
>> options VIMAGE
>> options ROUTETABLES=8
>> options RADIX_MPATH
>>
>> options SW_WATCHDOG
>>
>> device crypto
>> device cryptodev
>> device glxsb
>>
>> options BOOTVERBOSE=1
>>
>> #device pf
>> #device pflog
>> #device pfsync
>> device carp
>> device enc
>> device lagg
>> device epair
>>
>> #options ALTQ
>> #options ALTQ_CBQ
>> #options ALTQ_RED
>> #options ALTQ_RIO
>> #options ALTQ_HFSC
>> #options ALTQ_PRIQ
>>
>> options IPFIREWALL
>> options IPFIREWALL_DEFAULT_TO_ACCEPT
>> options IPFIREWALL_NAT
>> options LIBALIAS
>> options IPDIVERT
>> options DUMMYNET
>>
>> device bpf # Berkeley packet filter
>>
>> # USB support
>> options USB_DEBUG # enable debug msgs
>> device uhci # UHCI PCI->USB interface
>> device ohci # OHCI PCI->USB interface
>> device ehci # EHCI PCI->USB interface (USB 2.0)
>> device usb # USB Bus (required)
>> device umass # Disks/Mass storage - Requires scbus and da
>>
>>
>> Also src.conf and make.conf :
>>
>> root at vpn_vrf:[VNET(x)]:/usr/src/sys # cat /etc/src.conf
>> WITHOUT_ACCT=yes
>> WITHOUT_ACPI=yes
>> WITHOUT_AMD=yes
>> WITHOUT_APM=yes
>> WITHOUT_ASSERT_DEBUG=yes
>> WITHOUT_AT=yes
>> WITHOUT_ATF=yes
>> WITHOUT_ATM=yes
>> WITHOUT_AUDIT=yes
>> WITHOUT_BLUETOOTH=yes
>> WITHOUT_CALENDAR=yes
>> WITHOUT_CDDL=yes
>> WITHOUT_CTM=yes
>> WITHOUT_DICT=yes
>> WITHOUT_FLOPPY=yes
>> WITHOUT_GAMES=yes
>> WITHOUT_HTML=yes
>> WITHOUT_INFO=yes
>> WITHOUT_IPFILTER=yes
>> WITHOUT_IPX=yes
>> #WITHOUT_KERNEL_SYMBOLS=yes
>> WITHOUT_LEGACY_CONSOLE=yes
>> WITHOUT_LOCALES=yes
>> WITHOUT_LPR=yes
>> WITHOUT_MAIL=yes
>> WITHOUT_NDIS=yes
>> WITHOUT_QUOTAS=yes
>> WITHOUT_ROUTED=yes
>> WITHOUT_SENDMAIL=yes
>> WITH_SVN=yes
>> WITHOUT_ZFS=yes
>>
>> root at vpn_vrf:[VNET(x)]:/usr/src/sys # cat /etc/make.conf
>> CFLAGS=-O2
>> COPTFLAGS= -O -pipe
>> CPUTYPE=geode
>> KERNCONF=MARS
>> NO_MODULES=yes
>> BOOTWAIT=0
>> DOC_LANG=en_US.ISO8859-1
>>
>>
>>
>> --Nikolay
>>
>
> Also, originally I thought that the panic is when a multi path route is
> being deleted, however again from the coredump it seems that the panic
> happens when openvpn deletes the host route it installs for the remote
> openvpn server pointed to the default gw (before openvpn installs the new
> default gw pointing to the vpn tunnel) :
>
> (kgdb) fr 12
> (kgdb) x/12sb td->td_proc->p_args
> 0xc4269780: "\001"
> 0xc4269782: ""
> 0xc4269783: ""
> 0xc4269784: "B"
> 0xc4269786: ""
> 0xc4269787: ""
> 0xc4269788: "/sbin/route"
> 0xc4269794: "delete"
> 0xc426979b: "-net"
> 0xc42697a0: "78.90.222.xxx"
> 0xc42697ad: "10.255.255.0"
> 0xc42697ba: "255.255.255.255"
> (kgdb)
>
> I'm trying to reproduce this on a VirtualBox instance now, however so far no
> luck (no OpenVPN running, just adding and removing routes).
>
>
> --Nikolay
More information about the freebsd-net
mailing list