From nobody Tue Aug 02 06:11:10 2022 X-Original-To: bugs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Lxl3y1d5Cz4YGYt for ; Tue, 2 Aug 2022 06:11:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Lxl3x6kYDz3s4c for ; Tue, 2 Aug 2022 06:11:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Lxl3x5qb9z12N2 for ; Tue, 2 Aug 2022 06:11:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 2726B9i0035329 for ; Tue, 2 Aug 2022 06:11:09 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 2726B9ed035328 for bugs@FreeBSD.org; Tue, 2 Aug 2022 06:11:09 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 265569] [panic] Fatal trap 9: general protection fault while in kernel mode arc_reap Date: Tue, 02 Aug 2022 06:11:10 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: postmaster@willardstanley.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Bug reports List-Archive: https://lists.freebsd.org/archives/freebsd-bugs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-bugs@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1659420669; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=sdlCuAOuAdrzWjM+nwdazYu0iJrO8SUuAQJ5JRQ+XOs=; b=gSpJn4u/Z7ESoPVrdnlvTtVfUko1e99SAoujtuOpiAmZ0PK9EUKqM1FcGPLVFVQf8ARFr6 vmP/J65wN1uJCufG8Rr8Nb1/sLRW5S3eUfdjkvKvugYi5ojNrHVBhrJ+HMqSfmJCUsJs5a Cyc5aPEf/IKxKSOKUFimZYAQa1Tg0uCCkYw1qF8jhHpgEd+oApsU/ZEgCNrAyzZK5eF/wV sBYbaRHAdIegN5FfbVpST+IjQUmldxyYoEUUkbI4t0MztFOQf1PRgS+Rwdv2t221wjzajA iWWDlP/yXhCGlrA7uXYGwA3hw2AfnRqm/iLmbnWHKlk/uVbObGhHZdBS4TkD0g== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1659420669; a=rsa-sha256; cv=none; b=Lz+I0seZSqKHIIwquyyZw4wfh2Qn7C3tc4ur3/TrvsIKnKM1tSuPKjfmXaIBCXfCy5Mm4R E6qO7jtmeaCvcqIAypyuX7asjRAwszDaUdPp5WeUtR0b3Zj2MyAMlVkOyB8imRbCeA5jNV E/NmgZt7LnHJGLzAAH3M7pSGsm1SbBX9RAWRFTROxznQ0Alo3t1gczgZPCiyKLRxxckZAt BrwSiCLvM3Q/cl2ygsYCahwrJHu0uRMoYxa5TUZYLtmycV2Cw1GMejHXT0E7ZFcRpckhOO +8wMOls3zA9AZAeId+N3jQmYhfDrJJhaLdi8NZErz/sIot78Qw44I8lzhFuReQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D265569 Bug ID: 265569 Summary: [panic] Fatal trap 9: general protection fault while in kernel mode arc_reap Product: Base System Version: 13.1-RELEASE Hardware: amd64 OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: postmaster@willardstanley.com I can consistently trigger this panic on my system by running a userspace backup program (duplicity). The program seems to get to roughly the same pl= ace every time and then panics. I am going guess from the stack trace that perh= aps my system may have L2ARC corruption? I will disable the L2ARC and see if the problem goes away, but first I wanted to give the opportunity to further investigation. Other possibly interesting information about my system: Arc is a pair of nvme drives that are completely full: capacity operations bandwidth=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20 pool alloc free read write read write=20=20=20= =20=20=20=20=20=20=20=20=20=20=20 ------------------------ ----- ----- ----- ----- ----- -----=20=20=20= =20=20=20=20=20=20=20=20=20=20=20 spinning 13.0T 30.5T 2 0 26.9K 4.11K=20=20=20= =20=20=20=20=20=20=20=20=20=20=20 raidz1-0 13.0T 30.5T 2 0 26.7K 3.71K=20=20=20= =20=20=20=20=20=20=20=20=20=20=20 diskid/DISK-ZJV3CN53 - - 0 0 10.2K 982=20=20=20= =20=20=20=20=20=20=20=20=20=20=20 diskid/DISK-ZJV3CTT4 - - 0 0 3.20K 945=20=20=20= =20=20=20=20=20=20=20=20=20=20=20 diskid/DISK-ZJV3BQLS - - 0 0 9.35K 959=20=20=20= =20=20=20=20=20=20=20=20=20=20=20 diskid/DISK-ZJV3CM07 - - 0 0 3.99K 911=20=20=20= =20=20=20=20=20=20=20=20=20=20=20 logs - - - - - -=20=20=20= =20=20=20=20=20=20=20=20=20=20=20 mirror-1 1M 15.5G 0 0 250 413 nvd0p3 - - 0 0 125 206 nvd1p3 - - 0 0 125 206 cache - - - - - - nvd0p5 209G 91.8M 0 0 10.1K 12.0K nvd1p5 209G 35.7M 0 0 10.3K 10.6K ------------------------ ----- ----- ----- ----- ----- ----- Unread portion of the kernel message buffer: Fatal trap 9: general protection fault while in kernel mode cpuid =3D 1; apic id =3D 02 instruction pointer =3D 0x20:0xffffffff80f46649 stack pointer =3D 0x28:0xfffffe015668ad70 frame pointer =3D 0x28:0xfffffe015668adb0 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 5 (arc_reap) trap number =3D 9 panic: general protection fault cpuid =3D 1 time =3D 1659411380 KDB: stack backtrace: #0 0xffffffff80c69465 at kdb_backtrace+0x65 #1 0xffffffff80c1bb1f at vpanic+0x17f #2 0xffffffff80c1b993 at panic+0x43 #3 0xffffffff810afdf5 at trap_fatal+0x385 #4 0xffffffff81087528 at calltrap+0x8 #5 0xffffffff80f4b70a at bucket_drain+0xda #6 0xffffffff80f4ba7a at bucket_cache_reclaim_domain+0x2da #7 0xffffffff80f49412 at zone_reclaim+0x192 #8 0xffffffff821a2e59 at arc_reap_cb+0x9 #9 0xffffffff8231121a at zthr_procedure+0xba #10 0xffffffff80bd8a5e at fork_exit+0x7e #11 0xffffffff8108859e at fork_trampoline+0xe Uptime: 1h10m8s Dumping 5378 out of 65381 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..= 91% bt__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 55 __asm("movq %%gs:%P1,%0" : "=3Dr" (td) : "n" (offsetof(stru= ct pcpu, (kgdb) bt #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 #1 doadump (textdump=3D) at /usr/src/sys/kern/kern_shutdown= .c:399 #2 0xffffffff80c1b71c in kern_reboot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:487 #3 0xffffffff80c1bb8e in vpanic (fmt=3D0xffffffff811b4fb9 "%s", ap=3D) at /usr/src/sys/kern/kern_shutdown.c:920 #4 0xffffffff80c1b993 in panic (fmt=3D) at /usr/src/sys/kern/kern_shutdown.c:844 #5 0xffffffff810afdf5 in trap_fatal (frame=3D0xfffffe015668acb0, eva=3D0) = at /usr/src/sys/amd64/amd64/trap.c:944 #6 #7 0xffffffff80f46649 in slab_free_item (zone=3D0xfffffe015a7c8000, slab=3D0xfffff80e00000158, item=3D) at /usr/src/sys/vm/uma_core.c:4691 #8 zone_release (arg=3D0xfffffe015a7c8000, bucket=3D0xfffff80f4784e410, cnt=3D) at /usr/src/sys/vm/uma_core.c:4730 #9 0xffffffff80f4b70a in bucket_drain (zone=3Dzone@entry=3D0xfffffe015a7c8= 000, bucket=3Dbucket@entry=3D0xfffff80f4784e400) at /usr/src/sys/vm/uma_core.c:1= 312 #10 0xffffffff80f4ba7a in bucket_free (zone=3D0xfffffe015a7c8000, bucket=3D0xfffff80f4784e400, udata=3D0x0) at /usr/src/sys/vm/uma_core.c:520 #11 bucket_cache_reclaim_domain (zone=3Dzone@entry=3D0xfffffe015a7c8000, drain=3D, trim=3D, domain=3D, domain@entry=3D0) at /usr/src/sys/vm/uma_core.c:1509 #12 0xffffffff80f49412 in bucket_cache_reclaim (zone=3D0xfffffe015a7c8000, drain=3D, domain=3D-1) at /usr/src/sys/vm/uma_core.c:1534 #13 zone_reclaim (zone=3D0xfffffe015a7c8000, zone@entry=3D0xfffff80003fc580= 0, domain=3D, domain@entry=3D-1, waitok=3Dwaitok@entry=3D1, drain=3D) at /usr/src/sys/vm/uma_core.c:1674 #14 0xffffffff80f4927c in uma_zone_reclaim_domain (zone=3D, req=3D, req@entry=3D1, domain=3D-1) at /usr/src/sys/vm/uma_c= ore.c:5232 #15 0xffffffff82158f03 in kmem_cache_reap_soon (cache=3D) at /usr/src/sys/contrib/openzfs/module/os/freebsd/spl/spl_kmem.c:247 #16 0xffffffff82160451 in abd_cache_reap_now () at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/abd_os.c:508 #17 0xffffffff8219a69a in arc_kmem_reap_soon () at /usr/src/sys/contrib/openzfs/module/zfs/arc.c:4874 #18 0xffffffff821a2e59 in arc_reap_cb (arg=3D0xdffffb000, zthr=3D0xfffff80005f13a80) at /usr/src/sys/contrib/openzfs/module/zfs/arc.c= :5021 #19 0xffffffff8231121a in zthr_procedure (arg=3Darg@entry=3D0xfffff80003fc5= 800) at /usr/src/sys/contrib/openzfs/module/zfs/zthr.c:245 #20 0xffffffff80bd8a5e in fork_exit (callout=3D0xffffffff82311160 , arg=3D0xfffff80003fc5800, frame=3D0xfffffe015668af40) at /usr/src/sys/kern/kern_fork.c:1093 #21 #22 mi_startup () at /usr/src/sys/kern/init_main.c:322 #23 0xffffffff80f79189 in swapper () at /usr/src/sys/vm/vm_swapout.c:755 #24 0xffffffff80385022 in btext () at /usr/src/sys/amd64/amd64/locore.S:80 --=20 You are receiving this mail because: You are the assignee for the bug.=