From nobody Tue Jul 12 08:41:01 2022 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 3C4B11CFBE0B for ; Tue, 12 Jul 2022 08:41:18 +0000 (UTC) (envelope-from evgeniy@khramtsov.org) Received: from mxa.khramtsov.org (mxa.khramtsov.org [IPv6:2a0a:e5c0:2:10f::f]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4LhvNs3Rl4z3PK9; Tue, 12 Jul 2022 08:41:17 +0000 (UTC) (envelope-from evgeniy@khramtsov.org) Received: from mxa.khramtsov.org (mxa.khramtsov.org [IPv6:2a0a:e5c0:2:10f::f]) by mxa.khramtsov.org (Postfix) with ESMTP id 8CE13125EB7; Tue, 12 Jul 2022 08:33:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=khramtsov.org; s=rsa; t=1657614834; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=E4k1aw7i8YfVHDV+FyOOZaOM9oWQjr1oLunib2XJs0Y=; b=gcPvBdebb8lHnlUq9HCjFdTiRGkHiIYadP8RHOCAAjlu7dhDvddxIiLDyz8SGSeWo3Igvk MrP8pdJM4drSyFPjAEGxURhr0KGJ1hMdTNVv7/vzi+goR1QLHbHUY892/WxHfL0sWj3Gxt Brohz+2In0is+8S8qtWAPY32ORyJ7JUXTuERS2RHCEBWBg5lzLDbN4HSbtFUtmrOXUXVXi 0s/A7HP6mXwmzQ+bfkYmxKNocBTJQSUvX1kHSFvImU7N3RB7L+/afVds6LNhSmfddjtpB1 VZSotoMJ/K3JPTPyyHMPvOS0iDpV3m0FImz+BAvfxeQYxRPtKTDmyE5+m2OTEg== Date: Tue, 12 Jul 2022 08:41:01 +0000 From: Evgeniy Khramtsov To: Ryan Moeller Cc: FreeBSD-CURRENT@FreeBSD.org Subject: Re: BLAKE3 unstability? Message-ID: <20220712084101.iqvwyfuhge6myteq@vax.khramtsov.org> References: <20220709162640.7my2bq6rax5npdhf@vax.khramtsov.org> <20220709175605.ofkoft2mglrkaqpf@vax.khramtsov.org> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 4LhvNs3Rl4z3PK9 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=khramtsov.org header.s=rsa header.b=gcPvBdeb; dmarc=pass (policy=reject) header.from=khramtsov.org; spf=pass (mx1.freebsd.org: domain of evgeniy@khramtsov.org designates 2a0a:e5c0:2:10f::f as permitted sender) smtp.mailfrom=evgeniy@khramtsov.org X-Spamd-Result: default: False [-2.80 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[khramtsov.org,reject]; R_DKIM_ALLOW(-0.20)[khramtsov.org:s=rsa]; R_SPF_ALLOW(-0.20)[+ip6:2a0a:e5c0:2:10f::f:c]; RCVD_NO_TLS_LAST(0.10)[]; ONCE_RECEIVED(0.10)[]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[FreeBSD-CURRENT]; RCVD_COUNT_ONE(0.00)[1]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[khramtsov.org:+]; ASN(0.00)[asn:207996, ipnet:2a0a:e5c0:2::/48, country:CH]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; ARC_NA(0.00)[] X-ThisMailContainsUnwantedMimeParts: N > > > I can reproduce via: > > > > > > $ truncate -s 10G /tmp/test > > > $ mdconfig -f /tmp/test -S 4096 > > > $ zpool create test /dev/md1 > > > $ zfs create -o checksum=blake3 test/b > > > $ dd if=/dev/random of=/test/b/noise bs=1M count=4096 > > > $ sync > > > $ zpool scrub test > > > $ zpool status > > > > I cannot reproduce this on openzfs/zfs@cb01da68057 (the commit that was > > most recently merged) built out of tree on either stable/13 70fd40edb86 > > or main 9aa02d5120a. > > > > I'll update a system and see if I can reproduce it with the in-tree ZFS. > > > > - Ryan > > > It did not reproduce for me with in-tree ZFS on main@3c9ad9398fcd either. > > Could you share sysctl kstat.zfs.misc.chksum_bench, maybe we are using > different implementations? > I do see that blake3 went in with only a Linux module parameter for the > implementation selection, so I'll have to fix that. For now we can at least > see which was fastest, which should be the one selected. You just won't be > able to manually change it to see if that helps. > > - Ryan I found the culprit (kernel and base from download.FreeBSD.org kernel.txz and base.txz respectively) (I forgot about local sysctl.conf...): kern.sched.steal_thresh=1 kern.sched.preempt_thresh=121 Then #!/bin/sh truncate -s 10G /tmp/test mdconfig -f /tmp/test -S 4096 zpool create test /dev/md0 zfs create -o checksum=blake3 test/b dd if=/dev/random of=/test/b/noise bs=1M count=4096 sync zpool scrub test sleep 3 zpool status zpool destroy test mdconfig -d -u 0 rm /tmp/test As for ULE "tuning", these values give me fine desktop interactivity when building lang/rust when nice and idprio did not help, so I left them in sysctl.conf. Not sure if scheduling parameters are worthy of a ZFS PR, maybe something essential is preempted.