From nobody Tue Oct 17 16:40:37 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4S908z4dWCz4xk4n for ; Tue, 17 Oct 2023 16:40:51 +0000 (UTC) (envelope-from kevin.bowling@kev009.com) Received: from mail-pj1-x1035.google.com (mail-pj1-x1035.google.com [IPv6:2607:f8b0:4864:20::1035]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4S908z07p4z4PDl for ; Tue, 17 Oct 2023 16:40:50 +0000 (UTC) (envelope-from kevin.bowling@kev009.com) Authentication-Results: mx1.freebsd.org; dkim=none ("invalid DKIM record") header.d=kev009.com header.s=google header.b=VZvjlt8+; spf=pass (mx1.freebsd.org: domain of kevin.bowling@kev009.com designates 2607:f8b0:4864:20::1035 as permitted sender) smtp.mailfrom=kevin.bowling@kev009.com; dmarc=none Received: by mail-pj1-x1035.google.com with SMTP id 98e67ed59e1d1-27747002244so4703392a91.2 for ; Tue, 17 Oct 2023 09:40:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kev009.com; s=google; t=1697560849; x=1698165649; darn=freebsd.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=p2zxl/DXOrefDAqxTSJp+uKN/7M+CzuuMfn5ib13DGE=; b=VZvjlt8+a6cnHO1tWYaB0rtglIv5MNzd3SIByv9uEMYNzKsz+KLHgHQV9/m3wp/eLf 0qIt2XsqfyMPFmFmF4VrF1sSP+roHOOsvVA5V5vWJaLpBGfHyO1qG4JoytORypxu6jA1 edi9TwR7Olqo/HwKZaQl6au6gZr5jwpzm4JKY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697560849; x=1698165649; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=p2zxl/DXOrefDAqxTSJp+uKN/7M+CzuuMfn5ib13DGE=; b=fw5CUUKiiXi7tn/NOV7euSkZlOHARGQB3L4EicskauzMm5XXaG1FvmK989jv2YhwOc cN3/iQ2ARzfg5KklSiylCG0gqtBD8jFvHNeGq9bRIUzOs996ODMmYGmZQPC8hEI5jJEN qY3tjCQQMgY5LI9yg9IJZWTmXE/c+ludvffSOuzVZRjJwK+abnJeLq7dkhRbPzGJ0P7c gQsNzVhM5x08h2CEOMBk8TNwLIzHzlhC/XLnFLWhN0GZY6LhT1/+7q7juj7fmr4w8zSx dlM/ur2zuNQYWh/Y5boAvvChS2EXKCYIy2JP1hJUwC7jeaIVyg+3HzJvJs+oCvF82TY6 REtg== X-Gm-Message-State: AOJu0Yw1fgTHTQ/HZe48cVIF9EvJmbwklu4xRbY0oI4AC0EpCWoA5An/ /4AZcKnZPxtGyeUSXeeQlSf1v5h9ICAGnkT1eH6XHg== X-Google-Smtp-Source: AGHT+IFnpaMRPeXyIL2fP+n+5T6X1DP3hmbuzUHzn6Pc6WWktverVsJIHjH+N1omb9WO/WR0qfFCPA+GBTLhChTZwW8= X-Received: by 2002:a17:90b:a0c:b0:27d:5693:7340 with SMTP id gg12-20020a17090b0a0c00b0027d56937340mr2532803pjb.24.1697560849395; Tue, 17 Oct 2023 09:40:49 -0700 (PDT) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 From: Kevin Bowling Date: Tue, 17 Oct 2023 09:40:37 -0700 Message-ID: Subject: freebsd-update 12.3 to 14.0RC1 takes 12-24 hours (block cloning regression) To: freebsd-current@freebsd.org, freebsd-stable@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.30 / 15.00]; NEURAL_HAM_SHORT(-1.00)[-0.999]; NEURAL_HAM_LONG(-1.00)[-0.998]; NEURAL_HAM_MEDIUM(-1.00)[-0.998]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[text/plain]; R_DKIM_PERMFAIL(0.00)[kev009.com:s=google]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1035:from]; DKIM_TRACE(0.00)[kev009.com:~]; RCVD_TLS_LAST(0.00)[]; TO_DN_NONE(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DMARC_NA(0.00)[kev009.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4S908z07p4z4PDl Hi, I have two systems with a zpool 2x2 mirror on 7.2k RPM disks. One system also has a flash SLOG. The flash SLOG system took around 12 hours to complete freebsd-update from 13.2 to 14.0-RC1. The system without the SLOG took nearly 24 hours. This was the result of ~50k patches, and ~10k files from freebsd-update and a very pathological 'install' command performance. 'ps auxww | grep install': root 52225 0.0 0.0 12852 2504 0 D+ 20:55 0:00.00 install -S -o 0 -g 0 -m 0644 b6850914127c27fe192a41387f5cec04a1d927e6605ff09e8fd88dcd74fdec9d ///usr/src/sys/netgraph/ng_vlan.h root 68042 0.0 0.0 13580 3648 0 I+ 02:24 0:01.14 /bin/sh /usr/sbin/freebsd-update install root 69946 0.0 0.0 13580 3632 0 S+ 02:24 0:15.65 /bin/sh /usr/sbin/freebsd-update install 'control+t on freebsd-update': load: 0.16 cmd: install 97128 [tx->tx_sync_done_cv] 0.67r 0.00u 0.00s 0% 2440k mi_switch+0xc2 _cv_wait+0x113 txg_wait_synced_impl+0xb9 txg_wait_synced+0xb dmu_offset_next+0x77 zfs_holey+0x137 zfs_fre ebsd_ioctl+0x4f vn_generic_copy_file_range+0x64b kern_copy_file_range+0x327 sys_copy_file_range+0x78 amd64_syscall+0x10c fast_syscall_common+0xf8 I spoke with mjg about this and because my pools do not have block cloning enabled, copy_file_range turns into a massive pessimization in 'install'. He suggested a workaround of 'sysctl vfs.zfs.dmu_offset_next_sync=0' but we should probably sort this out for 14.0-RELEASE. Regards, Kevin