From nobody Mon Apr 17 18:51:05 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q0bjm4P9lz4531L for ; Mon, 17 Apr 2023 18:51:08 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oi1-x233.google.com (mail-oi1-x233.google.com [IPv6:2607:f8b0:4864:20::233]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q0bjm095Tz40M4; Mon, 17 Apr 2023 18:51:08 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=OU2AbQh6; spf=pass (mx1.freebsd.org: domain of mjguzik@gmail.com designates 2607:f8b0:4864:20::233 as permitted sender) smtp.mailfrom=mjguzik@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-oi1-x233.google.com with SMTP id j12so17460094oij.3; Mon, 17 Apr 2023 11:51:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681757466; x=1684349466; h=cc:to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=OOADMwcRUH/jyAP85Hg4yCRdIpLfv19E5UI8eNSNqt8=; b=OU2AbQh6U0JAAUToXORe4hM51bVjH9WXW8QzPEFde5A6iPEFA55k300C70wKnZllGa jpCXDPJVALr1lWM6HcsmS03LYMGMMsobWgEkAZjSntQSn/HvRa3vOQMVMWQ7sReUAzoP tCvXXgog233Rto4oLw3CNGRLwWCEt7d8u/S2qYfspdUwsYibUHr1ZK9tWX9KViADRikR nBl0Tgk/nGsQhd8S1q/RZ2bTaO6jFaUB3VfYr9tOQum3TA8Cq8tUc1KdF3m52IgWRJNy DOiZ6JLV/en2m6pF3ep9FZeQl9390VYHQh+tmgnha7U9qJPHsnAUnPccoCszQyecUQnh Mapw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681757466; x=1684349466; h=cc:to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=OOADMwcRUH/jyAP85Hg4yCRdIpLfv19E5UI8eNSNqt8=; b=Rhl/z/gZbBR7SfQzco3bCCsRIbk/4YaDlQPeeFgRM2I9EBPF6miS2tweh1IC2KSGjW WCsepgmDAfUgWGPI8JJHOkZMuT028OAOiS1HNXIoOt3p3lolUAG446yQsz3fSl7/rVA2 6f3N83ggPrgv74tNoOwwMPPL/6VgoIVy/C8wUNvAtZTT2ZCA8uPXUJxWjtRJ78Qpl6i6 cJpFDG0XQVmVaIChgTc/dbCC5pO6QCkBBFhW6Vx7i4uDW8vVTsVBIM4NMIv2ca0/Up6N t1Z2YJrRoo7ybJXFLgCfT3eE9Fu08tYHOOj2k6d/malhqvq/vvIjN28vUflBVaJkTPfr 8Svw== X-Gm-Message-State: AAQBX9duBVAExZMr9E9XnzQ4+oyTvoq2P8Z7XXImv+3VyxzhbnFsF8Rb BO2Lm2C9/OlvD9P9QO77XpG3wDZSr6XrdXOugaE3UsOR X-Google-Smtp-Source: AKy350YPSLTKgyBH0XWV/Bf7nsQn7EOhOvaw2fm2dDBM4ClQszOKTvETlVRcGMFpEs0nkQDI7gVExjuwAovPDAdWFpY= X-Received: by 2002:aca:f043:0:b0:38e:30:1225 with SMTP id o64-20020acaf043000000b0038e00301225mr961960oih.4.1681757466033; Mon, 17 Apr 2023 11:51:06 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Received: by 2002:a8a:46:0:b0:49c:b071:b1e3 with HTTP; Mon, 17 Apr 2023 11:51:05 -0700 (PDT) From: Mateusz Guzik Date: Mon, 17 Apr 2023 20:51:05 +0200 Message-ID: Subject: another crash and going forward with zfs To: freebsd-current@freebsd.org Cc: Pawel Jakub Dawidek , Glen Barber Content-Type: text/plain; charset="UTF-8" X-Spamd-Result: default: False [-3.97 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.97)[-0.973]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::233:from]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ARC_NA(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; MID_RHS_MATCH_FROMTLD(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4Q0bjm095Tz40M4 X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N After bugfixes got committed I decided to zpool upgrade and sysctl vfs.zfs.bclone_enabled=1 vs poudriere for testing purposes. I very quickly got a new crash: panic: VERIFY(arc_released(db->db_buf)) failed cpuid = 9 time = 1681755046 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0a90b8e5f0 vpanic() at vpanic+0x152/frame 0xfffffe0a90b8e640 spl_panic() at spl_panic+0x3a/frame 0xfffffe0a90b8e6a0 dbuf_redirty() at dbuf_redirty+0xbd/frame 0xfffffe0a90b8e6c0 dmu_buf_will_dirty_impl() at dmu_buf_will_dirty_impl+0xa2/frame 0xfffffe0a90b8e700 dmu_write_uio_dnode() at dmu_write_uio_dnode+0xe9/frame 0xfffffe0a90b8e780 dmu_write_uio_dbuf() at dmu_write_uio_dbuf+0x42/frame 0xfffffe0a90b8e7b0 zfs_write() at zfs_write+0x672/frame 0xfffffe0a90b8e960 zfs_freebsd_write() at zfs_freebsd_write+0x39/frame 0xfffffe0a90b8e980 VOP_WRITE_APV() at VOP_WRITE_APV+0xdb/frame 0xfffffe0a90b8ea90 vn_write() at vn_write+0x325/frame 0xfffffe0a90b8eb20 vn_io_fault_doio() at vn_io_fault_doio+0x43/frame 0xfffffe0a90b8eb80 vn_io_fault1() at vn_io_fault1+0x161/frame 0xfffffe0a90b8ecc0 vn_io_fault() at vn_io_fault+0x1b5/frame 0xfffffe0a90b8ed40 dofilewrite() at dofilewrite+0x81/frame 0xfffffe0a90b8ed90 sys_write() at sys_write+0xc0/frame 0xfffffe0a90b8ee00 amd64_syscall() at amd64_syscall+0x157/frame 0xfffffe0a90b8ef30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0a90b8ef30 --- syscall (4, FreeBSD ELF64, write), rip = 0x103cddf7949a, rsp = 0x103cdc85dd48, rbp = 0x103cdc85dd80 --- KDB: enter: panic [ thread pid 95000 tid 135035 ] Stopped at kdb_enter+0x32: movq $0,0x9e4153(%rip) The posted 14.0 schedule which plans to branch stable/14 on May 12 and one cannot bet on the feature getting beaten up into production shape by that time. Given whatever non-block_clonning and not even zfs bugs which are likely to come out I think this makes the feature a non-starter for said release. I note: 1. the current problems did not make it into stable branches. 2. there was block_cloning-related data corruption (fixed) and there may be more 3. there was unrelated data corruption (see https://github.com/openzfs/zfs/issues/14753), sorted out by reverting the problematic commit in FreeBSD, not yet sorted out upstream As such people's data may be partially hosed as is. Consequently the proposed plan is as follows: 1. whack the block cloning feature for the time being, but make sure pools which upgraded to it can be mounted read-only 2. run ztest and whatever other stress testing on FreeBSD, along with restoring openzfs CI -- I can do the first part, I'm sure pho will not mind to run some tests of his own 3. recommend people create new pools and restore data from backup. if restoring from backup is not an option, tar or cp (not zfs send) from the read-only mount block cloning beaten into shape would use block_cloning_v2 or whatever else, key point that the current feature name would be considered bogus (not blocking RO import though) to prevent RW usage of the current pools with it enabled. Comments? -- Mateusz Guzik