From nobody Sun Apr 09 21:58:04 2023 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PvmFB6WK2z44Kyw; Sun, 9 Apr 2023 21:58:06 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oi1-x22c.google.com (mail-oi1-x22c.google.com [IPv6:2607:f8b0:4864:20::22c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PvmFB4Tfrz3vT3; Sun, 9 Apr 2023 21:58:06 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-oi1-x22c.google.com with SMTP id bh10so2289837oib.1; Sun, 09 Apr 2023 14:58:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1681077485; x=1683669485; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=+U5TcYFAfdiJB7QwZpnQZkdWH63UlTtgCMJUZGH+fEQ=; b=UcyP3xIHh4AoZi9ZTtPum2djCcnJKfJl9sm5FcvVWMs2DFWCBDpK6Q0FSFXZznzQtS dDCBRMN5tPbD+3Ejl8BS9/YjqbY2LaflBxpatZbigIkI7NZIo8jk5Uw7swScVJ9Na7Om PtAmUF3nCtqKyR/b73bbngPsvGmC9/OdQiJsdDMhw/5o8MVuXYDTi4WwKG64tYOD8g7+ tni+AKV/GAM5USfiCOxD92827/biMMO7OM4xu7rihtvBT2QfwvpdG3G4eczjWHNVFJIZ FAmoumRNM3eGdHw3yN7FWxzUNOztozPoBdwd8JkuK88eR33Ipw4Jb1vkVzlcLT00DgPx 847w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1681077485; x=1683669485; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=+U5TcYFAfdiJB7QwZpnQZkdWH63UlTtgCMJUZGH+fEQ=; b=KV4RSq0ap4BmKNUy2H9qWGlPtyam8+MSnWFeTbXMeT6w5/8AZwaN+RWTqgafH+Bnd6 5BRqh4XfEQJIPeNwLozJhhEQLtulwwPR8udIV9JhBw8GKM4/CZhmZW0TJaqX16Hhtw+E HS3/L67amcZ4ZDZ436TlBPNbxLLb+qkKcm3RsN80qKFauy0VKiM4sM+vilXsKZ06EjTJ 0AvbNGJqOIJyNODpja6YvPcnWwlJHSxy5GW28VaMjdZo7zgJDk7fQ7MZoeGTZulFO1ny xJrQwU1PKG+UNKVyfXYOuClgncD6pIZejWpJnr5rebEbYg1bpUlgVsrl0RXDhjDuliB3 xVbg== X-Gm-Message-State: AAQBX9ehtvIibcKJ07Iz05lz/kOqVYi0wt+9V7r+b0aOyIl61VO3tXo0 5YreRt/Wh09iUlLkcvrcnGo980dHNTGJUaj0OLhHb/woA38= X-Google-Smtp-Source: AKy350aN81ly70ztFgGGzGdiQQ/zxBRokhMe4edQ569I/1PH+mikCX/qEFbyuTC0QB9vDlxFlgrhNpmdbLkFabhGCXQ= X-Received: by 2002:a05:6808:3a96:b0:389:53dc:cdf2 with SMTP id fb22-20020a0568083a9600b0038953dccdf2mr2278592oib.4.1681077485169; Sun, 09 Apr 2023 14:58:05 -0700 (PDT) List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org MIME-Version: 1.0 Received: by 2002:ac9:798d:0:b0:49c:b071:b1e3 with HTTP; Sun, 9 Apr 2023 14:58:04 -0700 (PDT) In-Reply-To: <20230409202650.49130b92@thor.intern.walstatt.dynvpn.de> References: <202304031513.333FD6qw014903@gitrepo.freebsd.org> <20230403231444.CF48911F@slippy.cwsent.com> <20230403232549.73E331A2@slippy.cwsent.com> <20230403235851.84C0467@slippy.cwsent.com> <20230404052811.DA2172C1@slippy.cwsent.com> <7c75b934-cb0a-b32e-bc19-b1e15e8cf3aa@freebsd.org> <20230409202650.49130b92@thor.intern.walstatt.dynvpn.de> From: Mateusz Guzik Date: Sun, 9 Apr 2023 23:58:04 +0200 Message-ID: Subject: Re: git: 2a58b312b62f - main - zfs: merge openzfs/zfs@431083f75 To: FreeBSD User Cc: Charlie Li , Cy Schubert , Rick Macklem , Martin Matuska , src-committers@freebsd.org, dev-commits-src-all@freebsd.org, dev-commits-src-main@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4PvmFB4Tfrz3vT3 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_RCPT(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On 4/9/23, FreeBSD User wrote: > Am Sun, 9 Apr 2023 13:23:05 -0400 > Charlie Li schrieb: > >> Mateusz Guzik wrote: >> > On 4/9/23, Charlie Li wrote: >> >> I've also started noticing random artefacts and malformed files whilst >> >> building packages with poudriere, causing all sorts of "exec format >> >> error"s, missing .so files due to corruption, data file corruption >> >> causing unintended failure modes, etc. All without block_cloning; >> >> enabling such causes a panic of its own when starting multiple builder >> >> jails at once. >> >> >> > >> > what's the panic? >> > >> manually typed out: >> >> panic: VERIFY(!zil_replaying(zilog, tx)) failed >> >> cpuid = 7 >> time = 1681060472 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame >> 0xfffffe02a05b28a0 >> vpanic() at vpanic+0x152/frame 0xfffffe02a05b28f0 >> spl_panic() at spl_panic+0x3a/frame 0xfffffe02a05b2950 >> zfs_log_clone_range() at zfs_log_clone_range+0x1db/frame >> 0xfffffe02a05b29e0 >> zfs_clone_range() at zfs_clone_range+0xae2/frame 0xfffffe02a05b2bc0 >> zfs_freebsd_copy_file_range() at zfs_freebsd_copy_file_range+0xff/frame >> 0xfffffe02a05b2c40 >> vn_copy_file_range() at vn_copy_file_range+0x115/frame 0xfffffe02a05b2ce0 >> kern_copy_file_range() at kern_copy_file_range+0x34e/frame >> 0xfffffe02a05b2db0 >> sys_copy_file_range() at sys_copy_file_range+0x78/frame >> 0xfffffe02a05b2e00 >> amd64_syscall() at amd64_syscall+0x148/frame 0xfffffe02a05b2f30 >> fast_syscall_common() at fast_syscall_common+0xf8/frame >> 0xfffffe02a05b2f30 >> --- syscall (569, FreeBSD ELF64, copy_file_range), rip = 0x908d2a, rsp = >> 0x820c28e68, rbp = 0x820c292b0 --- >> KDB: enter: panic >> [ thread pid 1856 tid 102129 ] >> Stopped at kdb_enter+0x32: movq $0,0x12760f3(%rip) >> db> >> > > I have the same issue (crash on access of several, but random datasets). > > It started with /usr/ports build failures when performing updates or > rebuilding ports, > poudriere host doesn't work anymore, as soon as started building ports, the > hosts (several of > them, same OS revision, new ZFS option enabled) crash. > Also when building binaries for an pkg OS distribution. > > That host also reports a ZFS RAIDZ pool as corrupted, out of the blue! Some > files from a > poudriere build and /usr/ports build seem to have issues with some > temporarily created files > in work directory. > > On another host /usr/ports is residing on ZFS and it crashes also when > building/updating ports > (/usr/ports residing on ZFS) - but on the same host /home is also residing > on ZFS, but even > downloading large amounts of emails, the host seem to be stable. Have not > found out yet what > kind of file access triggers the crash. > I reproduced the VERIFY(!zil_replaying(zilog, tx)) panic. As the backtrace shows it triggers when using copy_file_range, I temporarily patched the kernel to never do block cloning. So far the only package which failed to build was sqlite and it was for a legitimate reason (compiler errored out due to a problem in the code). -- Mateusz Guzik