From nobody Wed Nov 27 00:01:51 2024 X-Original-To: bugs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XyfkS0N1Hz5fDJp for ; Wed, 27 Nov 2024 00:01:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XyfkR4pRbz4S4G for ; Wed, 27 Nov 2024 00:01:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1732665711; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=rupvJHABYlRLsvUQsSLRD2N+MnRQ+8yhDhtZS28wNjY=; b=UPsbmk3EjMXiSp2MZu5HvwI+vjrn50gKNzLdXqa6+lSh4oRCin6GN/b4oHi40fu0Y22KCf QOwjY4U9nXVZiR8l4j5LrZ9y7qf+mA3/ysUHrYCM978x7Zx0uCyOJJwlt9GsxaRarATxdy sR4GkTGit2bileQzVzir+fzNyIvOdud58domQckeZiWW8UAlHrNjj1JYIqtF6xYtj1xJAY vkVgwKkY6eF98eEpYz1uj4yUroYGN7EiNkZSzzOF1Sl6+RMaHlxhzzjmMH3J1x6KunGRgb oYMSltz/yRcLgNZmC0oFquu7/0Kf6A/IQ6JqmuisLN0Jmr+E/DuhlgFn+cboaA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1732665711; a=rsa-sha256; cv=none; b=JWidvh0OQpdy2rRIUsdxEj47u2qJMg76/iRhewoggUnN/5XtvgJs++aAygZeUHHW0mmlFt QNAQCQSkF26LjCQytpy8NKcf9v1VPksqzkN/fvWSQMt3vkb5HsIFd8YDyb68GuTATYcac+ y3T66aYIj6u/hg8m0ASpeyCa6QJeqGzYUbbMAQElFqApVI/1mwel6Y5KEqKhxOqCVInI/f 8/1e5zMa3VbpEl5pFP0foMq07BMI73LWpmLi0cMynAknkYAFNIbWmWyC2uJKu8iE3RiTT9 UWhBI3xPjFy3Tx5/TlvNY97Pjq7IWdi8mmhBEhA5Xj8tvTgtmz8B2qhOkwNZFQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4XyfkR4QsQzmh2 for ; Wed, 27 Nov 2024 00:01:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 4AR01pXX075258 for ; Wed, 27 Nov 2024 00:01:51 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 4AR01p6A075257 for bugs@FreeBSD.org; Wed, 27 Nov 2024 00:01:51 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 282994] Repeated kernel panics Date: Wed, 27 Nov 2024 00:01:51 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: jSML4ThWwBID69YC@protonmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Bug reports List-Archive: https://lists.freebsd.org/archives/freebsd-bugs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-bugs@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D282994 Bug ID: 282994 Summary: Repeated kernel panics Product: Base System Version: 14.2-RELEASE Hardware: amd64 OS: Any Status: New Severity: Affects Some People Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: jSML4ThWwBID69YC@protonmail.com Hello,=20 I've run into an issue where multiple FreeBSD 14.1-p6 and 14.2-RC1 servers = are crashing with a kernel panic and rebooting. This happens multiple times a d= ay on each server. Sometimes it takes a few hours, sometimes it crashes in min= utes of startup.=20 Please see the 14.2-RC1 dump below.=20 ---------------------------------------------------------------------------= ----- Reading symbols from /boot/kernel/kernel... Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug... Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode cpuid =3D 6; apic id =3D 12 fault virtual address =3D 0xd0 fault code =3D supervisor write data, page not present instruction pointer =3D 0x20:0xffffffff80bdc2f2 stack pointer =3D 0x28:0xfffffe03505a1d30 frame pointer =3D 0x28:0xfffffe03505a1d60 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 81514 (php) rdi: fffff8010000d900 rsi: fffff810d3d97210 rdx: fffff8150afe4c00 rcx: fffff8010000dd90 r8: 000000000000001a r9: 0000000000000000 rax: 0000000000000000 rbx: fffff80cc151c500 rbp: fffffe03505a1d60 r10: 0000000000000000 r11: 0000000000000001 r12: 0000000000000180 r13: 00000000f4cf7018 r14: fffff802fc69b800 r15: 0000000000000000 trap number =3D 12 panic: page fault cpuid =3D 6 time =3D 1732641372 KDB: stack backtrace: #0 0xffffffff80b8b89d at kdb_backtrace+0x5d #1 0xffffffff80b3dc01 at vpanic+0x131 #2 0xffffffff80b3dac3 at panic+0x43 #3 0xffffffff81025a0b at trap_fatal+0x40b #4 0xffffffff81025a56 at trap_pfault+0x46 #5 0xffffffff80ffc398 at calltrap+0x8 #6 0xffffffff80bdcbe3 at kern_shm_open2+0x443 #7 0xffffffff80bddc31 at sys_shm_open2+0x21 #8 0xffffffff810262c5 at amd64_syscall+0x115 #9 0xffffffff80ffccab at fast_syscall_common+0xf8 Uptime: 1h33m33s Dumping 8536 out of 163689 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%.= .91% Reading symbols from /boot/kernel/accf_http.ko... Reading symbols from /usr/lib/debug//boot/kernel/accf_http.ko.debug... Reading symbols from /boot/kernel/accf_data.ko... Reading symbols from /usr/lib/debug//boot/kernel/accf_data.ko.debug... Reading symbols from /boot/kernel/zfs.ko... Reading symbols from /usr/lib/debug//boot/kernel/zfs.ko.debug... Reading symbols from /boot/kernel/accf_dns.ko... Reading symbols from /usr/lib/debug//boot/kernel/accf_dns.ko.debug... Reading symbols from /boot/kernel/geom_mirror.ko... Reading symbols from /usr/lib/debug//boot/kernel/geom_mirror.ko.debug... Reading symbols from /boot/kernel/cryptodev.ko... Reading symbols from /usr/lib/debug//boot/kernel/cryptodev.ko.debug... Reading symbols from /boot/kernel/mac_seeotheruids.ko... Reading symbols from /usr/lib/debug//boot/kernel/mac_seeotheruids.ko.debug.= .. Reading symbols from /boot/kernel/coretemp.ko... Reading symbols from /usr/lib/debug//boot/kernel/coretemp.ko.debug... Reading symbols from /boot/kernel/mlx4en.ko... Reading symbols from /usr/lib/debug//boot/kernel/mlx4en.ko.debug... Reading symbols from /boot/kernel/mlx4.ko... Reading symbols from /usr/lib/debug//boot/kernel/mlx4.ko.debug... Reading symbols from /boot/kernel/pf.ko... Reading symbols from /usr/lib/debug//boot/kernel/pf.ko.debug... Reading symbols from /boot/kernel/fusefs.ko... Reading symbols from /usr/lib/debug//boot/kernel/fusefs.ko.debug... Reading symbols from /boot/kernel/acpi_wmi.ko... Reading symbols from /usr/lib/debug//boot/kernel/acpi_wmi.ko.debug... Reading symbols from /boot/kernel/if_lagg.ko... Reading symbols from /usr/lib/debug//boot/kernel/if_lagg.ko.debug... Reading symbols from /boot/kernel/if_infiniband.ko... Reading symbols from /usr/lib/debug//boot/kernel/if_infiniband.ko.debug... Reading symbols from /boot/kernel/if_bridge.ko... Reading symbols from /usr/lib/debug//boot/kernel/if_bridge.ko.debug... Reading symbols from /boot/kernel/bridgestp.ko... Reading symbols from /usr/lib/debug//boot/kernel/bridgestp.ko.debug... Reading symbols from /boot/kernel/mac_ntpd.ko... Reading symbols from /usr/lib/debug//boot/kernel/mac_ntpd.ko.debug... Reading symbols from /boot/kernel/nullfs.ko... --Type for more, q to quit, c to continue without paging-- Reading symbols from /usr/lib/debug//boot/kernel/nullfs.ko.debug... Reading symbols from /boot/kernel/if_epair.ko... Reading symbols from /usr/lib/debug//boot/kernel/if_epair.ko.debug... __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57 57 __asm("movq %%gs:%P1,%0" : "=3Dr" (td) : "n" (offsetof(stru= ct pcpu, (kgdb) frame 8 #8 shm_alloc (ucred=3D0xfffff80cc151c500, mode=3Dmode@entry=3D384, largepage=3D) at /usr/src/sys/kern/uipc_shm.c:954 954 obj->un_pager.swp.swp_priv =3D shmfd; (kgdb) p vfs_hash_tbl $1 =3D (struct vfs_hash_head *) 0xfffffe0109b12000 (kgdb) p mp No symbol "mp" in current context. (kgdb) p hash No symbol "hash" in current context. (kgdb) p *mp No symbol "mp" in current context. (kgdb)=20 ------------------ The servers this is happening on all have a few things in common.=20 1. They all access data storage over MooseFS.=20 2. They all use fusefs-libs3-3.16.2 and moosefs3-client-3.0.117_1 3. They all have many nullfs mounts. 5k+ 4: All user process are run inside standard thick jails.=20 5. They all use RCTL to control resources used by users. 6. They all run many hundreds of processes owned by different users. All the process access data over the network, including the data files which are distributed storage.=20 Please let me know if here is anything I can do to help. --=20 You are receiving this mail because: You are the assignee for the bug.=