From nobody Fri Jun 09 04:57:12 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Qcpj72bB3z4bcSn for ; Fri, 9 Jun 2023 04:57:15 +0000 (UTC) (envelope-from glebius@freebsd.org) Received: from glebi.us (glebi.us [162.251.186.162]) by mx1.freebsd.org (Postfix) with ESMTP id 4Qcpj63nprz4QJb; Fri, 9 Jun 2023 04:57:14 +0000 (UTC) (envelope-from glebius@freebsd.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=softfail (mx1.freebsd.org: 162.251.186.162 is neither permitted nor denied by domain of glebius@freebsd.org) smtp.mailfrom=glebius@freebsd.org; dmarc=none Received: by glebi.us (Postfix, from userid 1000) id 0516722D88; Thu, 8 Jun 2023 21:57:12 -0700 (PDT) Date: Thu, 8 Jun 2023 21:57:12 -0700 From: Gleb Smirnoff To: current@freebsd.org Cc: mm@freebsd.org Subject: Re: panic(s) in ZFS on CURRENT Message-ID: References: List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spamd-Result: default: False [0.50 / 15.00]; VIOLATED_DIRECT_SPF(3.50)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; RCVD_NO_TLS_LAST(0.10)[]; MLMMJ_DEST(0.00)[current@freebsd.org]; ASN(0.00)[asn:27348, ipnet:162.251.186.0/24, country:US]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[glebius]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DMARC_NA(0.00)[freebsd.org]; TO_DN_NONE(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4Qcpj63nprz4QJb X-Spamd-Bar: / X-ThisMailContainsUnwantedMimeParts: N On Thu, Jun 08, 2023 at 07:56:07PM -0700, Gleb Smirnoff wrote: T> I'm switching to INVARIANTS kernel right now and will see if that panics earlier. This is what I got with INVARIANTS: panic: VERIFY3(dev->l2ad_hand <= dev->l2ad_evict) failed (225142071296 <= 225142063104) cpuid = 17 time = 1686286015 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2c/frame 0xfffffe0160dcea90 kdb_backtrace() at kdb_backtrace+0x46/frame 0xfffffe0160dceb40 vpanic() at vpanic+0x21f/frame 0xfffffe0160dcebe0 spl_panic() at spl_panic+0x4d/frame 0xfffffe0160dcec60 l2arc_write_buffers() at l2arc_write_buffers+0xcda/frame 0xfffffe0160dcedf0 l2arc_feed_thread() at l2arc_feed_thread+0x547/frame 0xfffffe0160dceec0 fork_exit() at fork_exit+0x122/frame 0xfffffe0160dcef30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0160dcef30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Uptime: 1m4s Dumping 5473 out of 65308 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% (kgdb) frame 4 #4 0xffffffff804342ea in l2arc_write_buffers (spa=0xfffffe022e942000, dev=0xfffffe023116a000, target_sz=16777216) at /usr/src/FreeBSD/sys/contrib/openzfs/module/zfs/arc.c:9445 9445 ASSERT3U(dev->l2ad_hand, <=, dev->l2ad_evict); (kgdb) p dev $1 = (l2arc_dev_t *) 0xfffffe023116a000 (kgdb) p dev->l2ad_hand $2 = 225142071296 (kgdb) p dev->l2ad_evict $3 = 225142063104 (kgdb) p *dev value of type `l2arc_dev_t' requires 66136 bytes, which is more than max-value-size Never seen kgdb not being able to print a structure that reported to be too big. -- Gleb Smirnoff