From nobody Mon Feb 19 22:50:19 2024 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TdyRt1xNfz59fj1 for ; Mon, 19 Feb 2024 22:50:34 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pj1-x1036.google.com (mail-pj1-x1036.google.com [IPv6:2607:f8b0:4864:20::1036]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TdyRt0LL8z4ZHM for ; Mon, 19 Feb 2024 22:50:34 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pj1-x1036.google.com with SMTP id 98e67ed59e1d1-290fb65531eso2621343a91.2 for ; Mon, 19 Feb 2024 14:50:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708383031; x=1708987831; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=re9Pir0cMj+JUlBRyn8y/Vix6g2xxY9rbjMO6u4Yy+U=; b=U9BhIKAUABqylRSPHoBHFZAJcDhwQhep/u/5IdwkO5094eenv5yZ91f50tBiRUcccG od1sg1B4vuhiDtE+3+BlhEXQtQA+C21WoqYjOkxg6lSwsh14uhsuzPLicjiKHLPcY+G0 ztw2YrTWSucGXFznmrzSm46AYkbQdq3d+CAMg83Axpit+UfcMGodJfb435voqC3v+bsH loDoU1RuF7CUKjYhf/HBoopgpBlnFFVZe+rAqxmhURGawkZbgJ5ckH7tC++rA19EI99P 5bmFyaq9edSVzKnobDy629iU/cNynwpeHpgxCXz7OoNL286yzti5HBI4aUDdG8IiK73e wGwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708383031; x=1708987831; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=re9Pir0cMj+JUlBRyn8y/Vix6g2xxY9rbjMO6u4Yy+U=; b=qJ2gorcjcwHSQBjF52oZn74rEBXO2iww7oMxhAq9frnhHSB51fapojAzQ2grcvzbyP 3MyDhCwONGbthZO0ackKkKHhC5UbpEKAIyguc+386+DowOpkUM0/1FkzTvrjmDRoJ2nx KkUTfNzgcPN8EPENSqKP6Z4gAIA1C0jkXJJEG6miNX3+dQ0Qoya9RaSW1DGZ6Gvx+P3N wr4CdP/v9pi9bqxzVvdgv+TrBcQCNXgbhFCDaXRQeUMKIpvlEFWW9MUizIBT/oRvWUbM +UDCb8jm2sJ0SrJvYwPYVyAylqx2Y8jmBVY0dKIk6k1onIv4MHdrGshnR2Vnkf/4xL3c oqHA== X-Gm-Message-State: AOJu0YwbcJ8c/S0qvGlL1XaHGiqLqw3TjlInNwfgWkgUSG4R607/n+1H 8ALQGHlc7X0q5/zYfThrEvUxM8Wf4cYkAdXVxeXnrNJ8Gix7WFa4+EVU4nq0R0+iojI369mXhyc mtTQDxRDeNwbwGPtyoSBc2HNtUA== X-Google-Smtp-Source: AGHT+IG8JGfKsLXmIxVfh0cjSFadvhuY9Anfe/Y8HzjzJ19tR6SsLabKgBUFHHqK1Linp7pm8zrukE3cuLhYqryQj10= X-Received: by 2002:a17:90a:ee90:b0:299:564f:c7d4 with SMTP id i16-20020a17090aee9000b00299564fc7d4mr4716067pjz.20.1708383031486; Mon, 19 Feb 2024 14:50:31 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <3ea6d241-b9cc-4294-aef8-ae1c6d9d8161@dartmouth.edu> <53139ffd-3e42-4aaf-a523-b8f4dc8b29a9@dartmouth.edu> <04d1f2e1-021e-42fb-9732-94fa98fd05fc@dartmouth.edu> In-Reply-To: <04d1f2e1-021e-42fb-9732-94fa98fd05fc@dartmouth.edu> From: Rick Macklem Date: Mon, 19 Feb 2024 14:50:19 -0800 Message-ID: Subject: Re: FreeBSD panics possibly caused by nfs clients To: "Matthew L. Dailey" Cc: "freebsd-current@freebsd.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4TdyRt0LL8z4ZHM X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] On Mon, Feb 19, 2024 at 7:44=E2=80=AFAM Matthew L. Dailey wrote: > > Hi all, > > So I finally induced a panic on a "pure" ufs system - root and exported > filesystem were both ufs. So, I think this definitively rules out zfs as > a source of the issue. > > This panic was on 14.0p5 without debugging options, so the core may not > be helpful. The panic and backtrace are below in case they're > interesting to anyone. > > Next, I'm going to try a CURRENT kernel without debugging options > enabled just to see if I can finally induce a panic here. My other two > VMs running CURRENT with full debugging are still clanking along. You might want to set kern.kstack_pages=3D6 in /boot/loader.conf in these setups. I would normally expect double faults when a kernel stack is blown, but maybe there is a reason that you do now see that for a blown kernel stack. (The impact of increasing stack pages from 4->6 should be minimal.) rick > > -Matt > > [218716] Fatal trap 12: page fault while in kernel mode > [218716] cpuid =3D 4; apic id =3D 08 > [218716] fault virtual address =3D 0x100000017 > [218716] fault code =3D supervisor read data, page not presen= t > [218716] instruction pointer =3D 0x20:0xffffffff80e9165d > [218716] stack pointer =3D 0x28:0xfffffe010b5aa3b0 > [218716] frame pointer =3D 0x28:0xfffffe010b5aa400 > [218716] code segment =3D base 0x0, limit 0xfffff, type 0x1b > [218716] =3D DPL 0, pres 1, long 1, def32 0, gran = 1 > [218716] processor eflags =3D interrupt enabled, resume, IOPL =3D 0 > [218716] current process =3D 49575 (nfsd: service) > [218716] rdi: 00000000ffffffff rsi: fffff800038ec900 rdx: fffffe00d932600= 0 > [218716] rcx: 0000000000030eb0 r8: 0000000000000000 r9: fffffe010b5aa41= 0 > [218716] rax: 00000000008f0eb0 rbx: fffff8038ac4cd00 rbp: fffffe010b5aa40= 0 > [218716] r10: 0000000000000000 r11: 0000000000000000 r12: 000000000000000= 0 > [218716] r13: fffff80003647c00 r14: fffff802f9dced00 r15: fffff800038ec90= 0 > [218716] trap number =3D 12 > [218716] panic: page fault > [218716] cpuid =3D 4 > [218716] time =3D 1708319487 > [218716] KDB: stack backtrace: > [218716] #0 0xffffffff80b9309d at kdb_backtrace+0x5d > [218716] #1 0xffffffff80b461a2 at vpanic+0x132 > [218716] #2 0xffffffff80b46063 at panic+0x43 > [218716] #3 0xffffffff8101d85c at trap_fatal+0x40c > [218716] #4 0xffffffff8101d8af at trap_pfault+0x4f > [218716] #5 0xffffffff80ff3fe8 at calltrap+0x8 > [218716] #6 0xffffffff80e8716e at newdirrem+0x8be > [218716] #7 0xffffffff80e866fa at softdep_setup_remove+0x1a > [218716] #8 0xffffffff80ea71af at ufs_dirremove+0x21f > [218716] #9 0xffffffff80ead4f4 at ufs_remove+0xb4 > [218716] #10 0xffffffff810f1428 at VOP_REMOVE_APV+0x28 > [218716] #11 0xffffffff80a60db4 at nfsvno_removesub+0xc4 > [218716] #12 0xffffffff80a52699 at nfsrvd_remove+0x1b9 > [218716] #13 0xffffffff80a374d4 at nfsrvd_dorpc+0x1854 > [218716] #14 0xffffffff80a4e76f at nfssvc_program+0x82f > [218716] #15 0xffffffff80e34080 at svc_run_internal+0xb50 > [218716] #16 0xffffffff80e3475b at svc_thread_start+0xb > [218716] #17 0xffffffff80b00b7f at fork_exit+0x7f > [218716] Uptime: 2d12h45m16s > [218716] Dumping 985 out of 16350 > MB:..2%..12%..22%..31%..41%..51%..61%..72%..82%..91% > > > #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57 > #1 doadump (textdump=3D) at > /usr/src/sys/kern/kern_shutdown.c:405 > #2 0xffffffff80b45d37 in kern_reboot (howto=3D260) > at /usr/src/sys/kern/kern_shutdown.c:526 > #3 0xffffffff80b4620f in vpanic (fmt=3D0xffffffff81147c9c "%s", > ap=3Dap@entry=3D0xfffffe010b5aa200) at > /usr/src/sys/kern/kern_shutdown.c:970 > #4 0xffffffff80b46063 in panic (fmt=3D) > at /usr/src/sys/kern/kern_shutdown.c:894 > #5 0xffffffff8101d85c in trap_fatal (frame=3D0xfffffe010b5aa2f0, > eva=3D4294967319) > at /usr/src/sys/amd64/amd64/trap.c:952 > #6 0xffffffff8101d8af in trap_pfault (frame=3D0xfffffe010b5aa2f0, > usermode=3Dfalse, signo=3D, ucode=3D) > at /usr/src/sys/amd64/amd64/trap.c:760 > #7 > #8 cancel_diradd (dap=3D0xfffff8038ac4cd00, > dirrem=3Ddirrem@entry=3D0xfffff800038ec900, > jremref=3Djremref@entry=3D0xfffff802f9dced00, > dotremref=3Ddotremref@entry=3D0x0, > dotdotremref=3Ddotdotremref@entry=3D0x0) > at /usr/src/sys/ufs/ffs/ffs_softdep.c:9028 > #9 0xffffffff80e8716e in newdirrem (bp=3D, > dp=3Ddp@entry=3D0xfffff800037fea80, ip=3Dip@entry=3D0xfffff8006b3b93= 00, > isrmdir=3Disrmdir@entry=3D0, > prevdirremp=3Dprevdirremp@entry=3D0xfffffe010b5aa4b0) > at /usr/src/sys/ufs/ffs/ffs_softdep.c:9480 > #10 0xffffffff80e866fa in softdep_setup_remove (bp=3D0xffffffff, > dp=3D0xfffff800038ec900, dp@entry=3D0xfffff800037fea80, > ip=3D0xfffffe00d9326000, > ip@entry=3D0xfffff8006b3b9300, isrmdir=3D200368, isrmdir@entry=3D0) > at /usr/src/sys/ufs/ffs/ffs_softdep.c:9176 > #11 0xffffffff80ea71af in ufs_dirremove (dvp=3Ddvp@entry=3D0xfffff801f764= be00, > ip=3Dip@entry=3D0xfffff8006b3b9300, flags=3D, > isrmdir=3Disrmdir@entry=3D0) at /usr/src/sys/ufs/ufs/ufs_lookup.c:11= 98 > #12 0xffffffff80ead4f4 in ufs_remove (ap=3D0xfffffe010b5aa5d8) > at /usr/src/sys/ufs/ufs/ufs_vnops.c:1054 > #13 0xffffffff810f1428 in VOP_REMOVE_APV ( > vop=3D0xffffffff8172f2d0 , a=3Da@entry=3D0xfffffe010b= 5aa5d8) > at vnode_if.c:1534 > #14 0xffffffff80a60db4 in VOP_REMOVE (dvp=3D0x8f0eb0, vp=3D0xfffff800539b= 7380, > cnp=3D0x30eb0) at ./vnode_if.h:789 > #15 nfsvno_removesub (ndp=3D0xfffffe010b5aa858, is_v4=3D, > cred=3D, p=3Dp@entry=3D0xfffffe010ae803a0, > exp=3Dexp@entry=3D0xfffffe010b5aaa88) > at /usr/src/sys/fs/nfsserver/nfs_nfsdport.c:1466 > #16 0xffffffff80a52699 in nfsrvd_remove (nd=3D0xfffffe010b5aabf8, > isdgram=3D, dp=3D, exp=3D0xfffffe010b5= aaa88) > at /usr/src/sys/fs/nfsserver/nfs_nfsdserv.c:1583 > #17 0xffffffff80a374d4 in nfsrvd_compound (nd=3D0xfffffe010b5aabf8, isdgr= am=3D0, > tag=3D, taglen=3D0, minorvers=3D) > at /usr/src/sys/fs/nfsserver/nfs_nfsdsocket.c:1322 > #18 nfsrvd_dorpc (nd=3Dnd@entry=3D0xfffffe010b5aabf8, isdgram=3Disdgram@e= ntry=3D0, > tag=3D, taglen=3D0, minorvers=3D) > at /usr/src/sys/fs/nfsserver/nfs_nfsdsocket.c:633 > #19 0xffffffff80a4e76f in nfs_proc (nd=3D0xfffffe010b5aabf8, > xid=3D, xprt=3D0xfffff800037b4800, rpp=3D) > at /usr/src/sys/fs/nfsserver/nfs_nfsdkrpc.c:464 > #20 nfssvc_program (rqst=3D0xfffff80128ac9000, xprt=3D0xfffff800037b4800) > at /usr/src/sys/fs/nfsserver/nfs_nfsdkrpc.c:348 > #21 0xffffffff80e34080 in svc_executereq (rqstp=3D0xfffff80128ac9000) > at /usr/src/sys/rpc/svc.c:1036 > #22 svc_run_internal (grp=3Dgrp@entry=3D0xfffff800033c2100, > ismaster=3Dismaster@entry=3D0) at /usr/src/sys/rpc/svc.c:1312 > #23 0xffffffff80e3475b in svc_thread_start (arg=3D0xffffffff, > arg@entry=3D0xfffff800033c2100) at /usr/src/sys/rpc/svc.c:1340 > #24 0xffffffff80b00b7f in fork_exit ( > callout=3D0xffffffff80e34750 , arg=3D0xfffff800033= c2100, > frame=3D0xfffffe010b5aaf40) at /usr/src/sys/kern/kern_fork.c:1160 > #25 > #26 0x000033af7af143ea in ?? ()