From nobody Tue May 30 23:16:09 2023 X-Original-To: questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QW7Yy0khMz4YKSK for ; Tue, 30 May 2023 23:16:22 +0000 (UTC) (envelope-from dpchrist@holgerdanske.com) Received: from holgerdanske.com (holgerdanske.com [184.105.128.27]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "holgerdanske.com", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QW7Yw6Lyfz3Pq9 for ; Tue, 30 May 2023 23:16:20 +0000 (UTC) (envelope-from dpchrist@holgerdanske.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=holgerdanske.com header.s=nov-20210719-112354 header.b=HM9uW5Wk; spf=pass (mx1.freebsd.org: domain of dpchrist@holgerdanske.com designates 184.105.128.27 as permitted sender) smtp.mailfrom=dpchrist@holgerdanske.com; dmarc=pass (policy=none) header.from=holgerdanske.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=holgerdanske.com; s=nov-20210719-112354; t=1685488570; bh=AG+1Y4+9MY0aAnkuazSzZj8stnuQd9MLVylONB0ONtc=; h=Received:Message-ID:Date:MIME-Version:User-Agent:Subject:To: References:Content-Language:From:In-Reply-To:Content-Type: Content-Transfer-Encoding; b=HM9uW5WkiBRU/9DSnJL5ZD8qfu3HTBWyougm5cvaAE7bmHvWSGDcRtDxwBAzISAP3 N4A3JXZVWZMbgAAfiWiK+4v3Ja1g6psJjhlAYEBxU8TOSGc0yXnDDuT52w5Ti5hCZ0 Cs3CFs3xTlwcXMyo0UKygdCqKicqcHH6omJR1WAIP1Ofi9YpXSTYfsNfNsKRx2iVIA HnrpWvYviZ/qzRucwLNmLIrRvmNWf0y3nObRG4ga/P2JAcTpC8qkuXrfVbytInOVQx SODTsDYsumy502Xi3SWszL8fmm3I8fPWf4p9/wx8mVGXap6ocPsyjVbNxOMyTSDUQm BeUYF3kaVRJpgPCllI9RKuc1xzxGrBMSZ5ZP470W9PNpw2WmpJjjSRNPbpgGC1oz8m amVR/iFQPVJCwZbPuZv8s7bWoYzeFcFCrbTGHyM93FGmmaiN2U9DokQZFbkEgnR0sP Jd8SmYx8rSbRaLtKju4yu46sVXlqpiXWGMK1zIEItry8xACYFxcJQHwjB3t/7ToheI iPpMvm9f1n5cPv9ZDDB+GPnNhjS2Uf1Hnv2qoy/zDiQyz8smcgcAALgnUnt/hYpO64 NfNJpQc9KW+TPEoHyJzfknXVMEu4B0Rk5iRedVlXuBTAUXj5nxNWkCYoz40zMiRb3H naBk00WRD39BWpnVIN7TVohk= Received: from 99.100.19.101 (99-100-19-101.lightspeed.frokca.sbcglobal.net [99.100.19.101]) by holgerdanske.com with ESMTPSA (TLS_AES_128_GCM_SHA256:TLSv1.3:Kx=any:Au=any:Enc=AESGCM(128):Mac=AEAD) (SMTP-AUTH username dpchrist@holgerdanske.com, mechanism PLAIN) for ; Tue, 30 May 2023 16:16:10 -0700 Message-ID: Date: Tue, 30 May 2023 16:16:09 -0700 List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.11.0 Subject: Re: read and write back full disk to trigger relocation To: questions@freebsd.org References: <1957739901.520492.1685310340560@ichabod.co-bxl> <0d0186c5-9542-1af3-2ce3-e28480b4b6d7@holgerdanske.com> <1961596841.3509648.1685359514813@fidget.co-bxl> <00671d49-83b1-26a0-4e28-47eb0d7cb95c@holgerdanske.com> <2025846914.656453.1685438293844@ichabod.co-bxl> Content-Language: en-US From: David Christensen In-Reply-To: <2025846914.656453.1685438293844@ichabod.co-bxl> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; DMARC_POLICY_ALLOW(-0.50)[holgerdanske.com,none]; R_DKIM_ALLOW(-0.20)[holgerdanske.com:s=nov-20210719-112354]; R_SPF_ALLOW(-0.20)[+a]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; BLOCKLISTDE_FAIL(0.00)[99.100.19.101:server fail,184.105.128.27:server fail]; ASN(0.00)[asn:6939, ipnet:184.104.0.0/15, country:US]; MLMMJ_DEST(0.00)[questions@freebsd.org]; DKIM_TRACE(0.00)[holgerdanske.com:+]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[questions@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4QW7Yw6Lyfz3Pq9 X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On 5/30/23 02:18, Sysadmin Lists wrote: > David Christensen May 29, 2023, 4:12:24 PM >> Testing dd(1) and gmirror(8): >> >> 2023-05-29 15:21:32 toor@vf1 ~ >> # freebsd-version ; uname -a >> 12.4-RELEASE-p2 >> FreeBSD vf1.tracy.holgerdanske.com 12.4-RELEASE-p1 FreeBSD >> 12.4-RELEASE-p1 GENERIC amd64 >> >> 2023-05-29 15:23:05 toor@vf1 ~ >> # gmirror label mymirror ada3 ada4 >> >> 2023-05-29 15:24:11 toor@vf1 ~ >> # gmirror status mymirror >> Name Status Components >> mirror/mymirror COMPLETE ada3 (ACTIVE) >> ada4 (ACTIVE) >> >> 2023-05-29 15:52:41 toor@vf1 ~ >> # dd if=/dev/ada3 of=/dev/ada3 bs=1m >> dd: /dev/ada3: Operation not permitted >> >> 2023-05-29 15:53:45 toor@vf1 ~ >> # dd if=/dev/ada4 of=/dev/ada4 bs=1m >> dd: /dev/ada4: Operation not permitted >> >> 2023-05-29 15:53:52 toor@vf1 ~ >> # dd if=/dev/mirror/mymirror of=/dev/mirror/mymirror bs=1m >> 1023+1 records in >> 1023+1 records out >> 1073741312 bytes transferred in 3.299006 secs (325474224 bytes/sec) >> >> >> This confirms that the kernel will not allow writes to mirror components >> when they are active, as it should. If a process could write to a >> component of a mirror, that would bypass the mirror driver, defeat the >> purpose of the mirror, allow race conditions, and result in data loss/ >> data corruption. > > That makes sense. I wouldn't recommend running it on a live system anyway. > Probably wiser to boot into a livecd and run it on a single disk. gmirror > shouldn't notice a difference since the data isn't presently corrupted, just > decaying (is my guess). 3TB is a lot of data to process. I also prefer to do disk maintenance activities when the disks are off-line, typically by booting alternate media (such as a live USB stick). I did the above testing on VirtualBox on Debian by creating two 1 GB virtual disks backed by files. When I created the virtual disks, I choose "Dynamic" sizing -- e.g. the backing files start small and grow as data is added. I have since noted that the size, mtime, and atime on the backing files have not changed since the files were created: 2023-05-30 15:56:36 dpchrist@taz ~/virtualbox/virtual-machines/vf1 $ stat vf1_?.vdi File: vf1_3.vdi Size: 2097152 Blocks: 16 IO Block: 4096 regular file Device: fe02h/65026d Inode: 392462 Links: 1 Access: (0600/-rw-------) Uid: (13250/dpchrist) Gid: (13250/dpchrist) Access: 2023-05-29 15:20:35.292781334 -0700 Modify: 2023-05-29 15:19:51.553228088 -0700 Change: 2023-05-29 15:19:51.553228088 -0700 Birth: 2023-05-29 15:13:28.182411743 -0700 File: vf1_4.vdi Size: 2097152 Blocks: 16 IO Block: 4096 regular file Device: fe02h/65026d Inode: 392466 Links: 1 Access: (0600/-rw-------) Uid: (13250/dpchrist) Gid: (13250/dpchrist) Access: 2023-05-29 15:20:35.292781334 -0700 Modify: 2023-05-29 15:19:51.553228088 -0700 Change: 2023-05-29 15:19:51.553228088 -0700 Birth: 2023-05-29 15:13:44.630780217 -0700 If I do the dd(1) command again with O_DIRECT: 2023-05-30 15:59:06 toor@vf1 ~ # dd if=/dev/mirror/mymirror of=/dev/mirror/mymirror bs=1m oflag=direct 1023+1 records in 1023+1 records out 1073741312 bytes transferred in 3.465168 secs (309867017 bytes/sec) The size, mtime, and atime still do not change: 2023-05-30 15:59:55 dpchrist@taz ~/virtualbox/virtual-machines/vf1 $ stat vf1_?.vdi File: vf1_3.vdi Size: 2097152 Blocks: 16 IO Block: 4096 regular file Device: fe02h/65026d Inode: 392462 Links: 1 Access: (0600/-rw-------) Uid: (13250/dpchrist) Gid: (13250/dpchrist) Access: 2023-05-29 15:20:35.292781334 -0700 Modify: 2023-05-29 15:19:51.553228088 -0700 Change: 2023-05-29 15:19:51.553228088 -0700 Birth: 2023-05-29 15:13:28.182411743 -0700 File: vf1_4.vdi Size: 2097152 Blocks: 16 IO Block: 4096 regular file Device: fe02h/65026d Inode: 392466 Links: 1 Access: (0600/-rw-------) Uid: (13250/dpchrist) Gid: (13250/dpchrist) Access: 2023-05-29 15:20:35.292781334 -0700 Modify: 2023-05-29 15:19:51.553228088 -0700 Change: 2023-05-29 15:19:51.553228088 -0700 Birth: 2023-05-29 15:13:44.630780217 -0700 So, either FreeBSD Or Virtual is optimizing away the write(2) calls because the write buffer matches what is already in a memory cache from prior read(2) calls (?). I would say the experiment should be repeated on real HDD's, but how do I detect if identical data has being written to the platters? The HDD controller also has a cache and could optimize away such writes. One idea would be to read into a buffer, invert the bits in the buffer, write the buffer, invert the bits again, and write again. These are the kinds of issues that the disk manufacturer is supposed to solve. Thus, my first response "I would look for a manufacturer diagnostic tool". David