From nobody Mon Aug 30 07:32:19 2021 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 981F21789E93 for ; Mon, 30 Aug 2021 07:32:21 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Gyhq93w3Tz4dBL; Mon, 30 Aug 2021 07:32:21 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from [192.168.0.88] (unknown [195.64.148.76]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: avg/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 1792D2D38B; Mon, 30 Aug 2021 07:32:20 +0000 (UTC) (envelope-from avg@FreeBSD.org) Subject: Re: network crash in nhop_free From: Andriy Gapon To: "Alexander V. Chernikov" Cc: freebsd-net References: <2fbc5205-3fcc-d233-dae1-cf6ddc8d691d@FreeBSD.org> <95F4F779-91A0-482B-B26B-6C95A60FC281@ipfw.ru> <70d1091d-07ec-1c76-29bc-1f2e2264b55a@FreeBSD.org> <869483A6-FA65-40A2-9CCC-05216588EAC8@ipfw.ru> <6a1874fc-a5ac-31a3-13ee-390177091ce6@FreeBSD.org> Message-ID: <24574d7e-6986-99ee-86f5-a1c1c513b3f6@FreeBSD.org> Date: Mon, 30 Aug 2021 10:32:19 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Firefox/78.0 Thunderbird/78.13.0 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 In-Reply-To: <6a1874fc-a5ac-31a3-13ee-390177091ce6@FreeBSD.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-ThisMailContainsUnwantedMimeParts: N On 30/08/2021 10:28, Andriy Gapon wrote: > On 01/08/2021 16:36, Alexander V. Chernikov wrote: >> >> >>> On 10 Jul 2021, at 10:07, Andriy Gapon wrote: >>> >>> On 09/07/2021 00:02, Alexander V. Chernikov wrote: >>>> Hi Andriy, >>>> Could you by any chance provide a bit more info on the system networking >>>> configuration and the steps leading to panic? >>>> No chance for a coredump? >>>> destroy_nhgrp() suggests that there was a multipath route (default?) that >>>> was deleted. >>>> nhops are created with UMA_ALIGN_PTR, so I suspect there is a garbage inside >>>> nhgrp pointer.. >>> >>> I've just reproduced the problem and got a crash dump. >>> The new panic is a little bit different, but I think that it confirms your >>> analysis. >>> Also, you are right about the multipath route, although its creation was not >>> intentional. >> >> Should be fixed by >> https://cgit.freebsd.org/src/commit/?id=054948bd81bb9e4e32449cf351b62e501b8831ff >> . > > I have to report that, unfortunately, as of main > bb958dcf3d8af3a033dacbf8133681c9b0c73b2f I can still reproduce the same panic > using the same steps. > To be clear, as I reported two similar but still distinct panics, it's the first > panic, "Misaligned access from kernel space!". > > I should also add that the commit message does not really match my scenario. > In my case routes do not change quite fast.  I have generous pauses between > starting and stopping ppp. > I have a feeling that there must something more deterministic that leads to the > crash. Some more details from the today's crash: panic: Misaligned access from kernel space! cpuid = 0 time = 1630308311 KDB: stack backtrace: db_trace_self() at db_trace_self db_trace_self_wrapper() at db_trace_self_wrapper+0x30 vpanic() at vpanic+0x184 panic() at panic+0x44 align_abort() at align_abort+0xb8 handle_el1h_sync() at handle_el1h_sync+0x78 --- exception, esr 0x96000021 nhop_free() at nhop_free+0x100 destroy_nhgrp() at destroy_nhgrp+0x38 epoch_call_task() at epoch_call_task+0x158 gtaskqueue_run_locked() at gtaskqueue_run_locked+0x178 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xc8 fork_exit() at fork_exit+0x74 fork_trampoline() at fork_trampoline+0x14 Uptime: 11m17s Dumping 150 out of 998 MB:..3%..11%..22%..32%..43%..51%..62%..72%..83% get_curthread () at /usr/devel/git/rock/sys/arm64/include/pcpu.h:68 68 /usr/devel/git/rock/sys/arm64/include/pcpu.h: No such file or directory. (kgdb) bt #0 get_curthread () at /usr/devel/git/rock/sys/arm64/include/pcpu.h:68 #1 doadump (textdump=textdump@entry=1) at /usr/devel/git/rock/sys/kern/kern_shutdown.c:417 #2 0xffff0000003bebf0 in kern_reboot (howto=260) at /usr/devel/git/rock/sys/kern/kern_shutdown.c:504 #3 0xffff0000003bf10c in vpanic (fmt=, ap=...) at /usr/devel/git/rock/sys/kern/kern_shutdown.c:947 #4 0xffff0000003bee3c in panic (fmt=0x0) at /usr/devel/git/rock/sys/kern/kern_shutdown.c:871 #5 0xffff0000006c2054 in align_abort (td=, frame=, esr=2516582433, far=16045693110842147062, lower=) at /usr/devel/git/rock/sys/arm64/arm64/trap.c:212 #6 #7 atomic_fetchadd_32_llsc (p=0xdeadc0dedeadc0f6, val=4294967295) at /usr/devel/git/rock/sys/arm64/include/atomic.h:316 #8 atomic_fetchadd_32 (p=, val=4294967295) at /usr/devel/git/rock/sys/arm64/include/atomic.h:316 #9 refcount_releasen (count=0xdeadc0dedeadc0f6, n=1) at /usr/devel/git/rock/sys/sys/refcount.h:152 #10 refcount_release (count=0xdeadc0dedeadc0f6) at /usr/devel/git/rock/sys/sys/refcount.h:174 #11 nhop_free (nh=) at /usr/devel/git/rock/sys/net/route/nhop_ctl.c:669 #12 0xffff000000506268 in free_nhgrp_nhops (nhg_priv=0xffffa00000d31b98) at /usr/devel/git/rock/sys/net/route/nhgrp_ctl.c:423 #13 destroy_nhgrp (nhg_priv=0xffffa00000d31b98) at /usr/devel/git/rock/sys/net/route/nhgrp_ctl.c:380 #14 0xffff000000405434 in epoch_call_task (arg=) at /usr/devel/git/rock/sys/kern/subr_epoch.c:819 #15 0xffff000000408ee4 in gtaskqueue_run_locked (queue=queue@entry=0xffffa00000c03c00) at /usr/devel/git/rock/sys/kern/subr_gtaskqueue.c:371 #16 0xffff000000408c38 in gtaskqueue_thread_loop (arg=arg@entry=0xffff000089b69008) at /usr/devel/git/rock/sys/kern/subr_gtaskqueue.c:547 #17 0xffff00000037701c in fork_exit (callout=0xffff000000408b6c , arg=0xffff000089b69008, frame=0xffff000087934990) at /usr/devel/git/rock/sys/kern/kern_fork.c:1087 -- Andriy Gapon