From nobody Sun Mar 03 05:25:18 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TnVds60c9z5C0P2 for ; Sun, 3 Mar 2024 05:25:21 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Received: from hergotha.csail.mit.edu (tunnel82308-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "garrett.wollman.name", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TnVds0dHLz499Q for ; Sun, 3 Mar 2024 05:25:21 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bimajority.org (policy=none); spf=pass (mx1.freebsd.org: domain of wollman@hergotha.csail.mit.edu designates 2001:470:1f06:ccb::2 as permitted sender) smtp.mailfrom=wollman@hergotha.csail.mit.edu Received: from hergotha.csail.mit.edu (localhost [127.0.0.1]) by hergotha.csail.mit.edu (8.17.1/8.17.1) with ESMTPS id 4235PJH0066202 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sun, 3 Mar 2024 00:25:19 -0500 (EST) (envelope-from wollman@hergotha.csail.mit.edu) Received: (from wollman@localhost) by hergotha.csail.mit.edu (8.17.1/8.17.1/Submit) id 4235PJFM066201; Sun, 3 Mar 2024 00:25:19 -0500 (EST) (envelope-from wollman) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <26084.2494.962383.278446@hergotha.csail.mit.edu> Date: Sun, 3 Mar 2024 00:25:18 -0500 From: Garrett Wollman To: stable@freebsd.org Cc: Rick Macklem Subject: Re: 13-stable NFS server hang In-Reply-To: <26083.64612.717082.366639@hergotha.csail.mit.edu> References: <26078.50375.679881.64018@hergotha.csail.mit.edu> <26083.64612.717082.366639@hergotha.csail.mit.edu> X-Mailer: VM 8.2.0b under 29.1 (amd64-portbld-freebsd13.2) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.4 (hergotha.csail.mit.edu [0.0.0.0]); Sun, 03 Mar 2024 00:25:19 -0500 (EST) X-Spam-Status: No, score=-4.5 required=5.0 tests=ALL_TRUSTED, HEADER_FROM_DIFFERENT_DOMAINS,NICE_REPLY_A,T_SCC_BODY_TEXT_LINE autolearn=disabled version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on hergotha.csail.mit.edu X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.68 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.996]; NEURAL_HAM_MEDIUM(-0.78)[-0.783]; FORGED_SENDER(0.30)[wollman@bimajority.org,wollman@hergotha.csail.mit.edu]; R_SPF_ALLOW(-0.20)[+ip6:2001:470:1f06:ccb::2]; DMARC_POLICY_SOFTFAIL(0.10)[bimajority.org : SPF not aligned (relaxed), No valid DKIM,none]; MIME_GOOD(-0.10)[text/plain]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEFALL_USER(0.00)[wollman]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; TAGGED_RCPT(0.00)[]; FREEMAIL_CC(0.00)[gmail.com]; R_DKIM_NA(0.00)[]; MLMMJ_DEST(0.00)[stable@freebsd.org]; FROM_NEQ_ENVFROM(0.00)[wollman@bimajority.org,wollman@hergotha.csail.mit.edu]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; ARC_NA(0.00)[] X-Rspamd-Queue-Id: 4TnVds0dHLz499Q < I believe this explains why vn_copy_file_range sometimes takes much > longer than a second: our servers often have lots of data waiting to > be written to disk, and if the file being copied was recently modified > (and so is dirty), this might take several seconds. I've set > vfs.zfs.dmu_offset_next_sync=0 on the server that was hurting the most > and am watching to see if we have more freezes. In case anyone is wondering why this is an issue, it's the combination of two factors: 1) vn_generic_copy_file_range() attempts to preserve holes in the source file. 2) ZFS does automatic hole-punching on write for filesystems where compression is enabled. It happens in the same code path as compression, checksum generation, and redundant-write suppression, and thus does not happen until the dirty blocks are about to be committed to disk. So if the file is dirty, ZFS doesn't "know" whether thare where the then-extant holes are until a sync has completed. While vn_generic_copy_file_range() has a flag to stop and return partial success after a second of copying, this flag does not affect sleeps internal to the filesystem, so zfs_holey() can sleep indefinitely and vn_generic_copy_file_range() can't do anything about it until the sync has already happened. -GAWollman