From nobody Sun Dec 11 15:08:55 2022 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NVSpF64X3z4kjQ0 for ; Sun, 11 Dec 2022 15:09:09 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ed1-x535.google.com (mail-ed1-x535.google.com [IPv6:2a00:1450:4864:20::535]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NVSpF4Lbmz4QDd for ; Sun, 11 Dec 2022 15:09:09 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ed1-x535.google.com with SMTP id d20so9604259edn.0 for ; Sun, 11 Dec 2022 07:09:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20210112.gappssmtp.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=GDVW9fgP0QE3kQoCPj2rUsU7fzWSlMY6l6XoHf7payE=; b=qWzjS4F2TFehW8EKGnZXK5FRClvphgre8IUh4ZHR+qnc9c/QYHQ80tqpyhLgMH54FB vnDqok5G38YLAhirDIYkBuhP+HebADiNeuR53i6lzxVzNuz+zdxPf5l+3F7M62iCSc5v oBLbrQfdv+LIrT2EyPpWJxTCjB0rHaTWaLjtdM+g6nQ1mKbwD5SSpbTq7fnSDDivE11u C/m+bwAUr7KMqN07R1dbfNO7CgFre2syA0VIKeeCpywMGPC+QdWkGtxp/+KYrHQDOHfc Dk14ht++00urRKf02q9CwWgtgudxWko5SkV2Ifp3Vo67FEAPFednkuxt+a+tj20nC6+e qL2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=GDVW9fgP0QE3kQoCPj2rUsU7fzWSlMY6l6XoHf7payE=; b=gjGpFsthS2mj8VOEmK79rqdHCH9JPRpF+KCcqq00IyT/HfNhz66BfekzErwN174VWy 5BXRIwKFQJ6vkn56xWljPm4bcQztr2riMjA//qphRfow5qB3+hKZu+XhaXawsKhEyxog bGqQMvJhqKMBi1TNvgDh4EgS5tmORprjwmhAI0UgmBKtejhpcCrubrn2E/bmETKJKd5D FrUpL2W0kxhIk8yxSjW4Xr5Y4qkrnl7TwmvUVhmocOfgc4gNdQ6NP5xkPnjI869x0HPP er69/svmwdl/rAenRbmssnLGyKsehhwa2bulzCq+Zxsrrvu/ruwqSQpXRldLEq0ES9Px ynGg== X-Gm-Message-State: ANoB5plDgGH1dz25FfoUeitmELSRvv/mcqXzBPMZzPNEtD/pmvoEI/ML Nc43xn4TDw5vbExuzCXnmz/ow9KFQEkG+R6HOZnuy76WaXiHe7HS X-Google-Smtp-Source: AA0mqf5rHOoAOkxAHj45LJqWFJX5VG2qb5CACA7duSDNQ6cDwhWCQS7p5E8nZgvM/7sW3tr0alC8UP27XopOUPKVfNg= X-Received: by 2002:aa7:d659:0:b0:46b:1687:2e5d with SMTP id v25-20020aa7d659000000b0046b16872e5dmr44833822edr.136.1670771346627; Sun, 11 Dec 2022 07:09:06 -0800 (PST) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: <85c5a64c-915e-d790-e617-c94f3fb7cd9a@gmail.com> In-Reply-To: From: Warner Losh Date: Sun, 11 Dec 2022 08:08:55 -0700 Message-ID: Subject: Re: Everchanging bytes at the end of mirror disks To: Artem Kuchin Cc: FreeBSD FS Content-Type: multipart/alternative; boundary="0000000000009df31c05ef8ec4c6" X-Rspamd-Queue-Id: 4NVSpF4Lbmz4QDd X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --0000000000009df31c05ef8ec4c6 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sun, Dec 11, 2022 at 1:45 AM Artem Kuchin wrote: > 11.12.2022 11:22, Warner Losh =D0=BF=D0=B8=D1=88=D0=B5=D1=82: > > > > On Sat, Dec 10, 2022, 11:52 PM Artem Kuchin > wrote: > >> Hello! >> >> I am writing a small utility for myseld and part of it is comparing >> gmirror disks. After running some tests i realized that some bytes at >> the very end of disks are constantly changing. >> > > The last sector has metadata about the mirror and about the mirror > element. It's this latter data that differs. > > > Thank you for reply. Then there are several question > > 1) Last SECTOR is not always 512KB or is it? Do i need to get block size > from diskinfo and subtract its size from disk size? > diskinfo(1) will tell you, it's returned with the DIOCGSECTORSIZE ioctl. > 2) Why its content is changing so often? On every write? How often? The > only place to look for description is the gmirror sources? > When a mirror breaks (that is, writes can happen to one side but not the other), we need to know right away which side is the more current one. The gmirror does this by modifying the metadata to record how many writes have happened to each mirror member (one reason that write is so expensive). > It does not look good to me, but maybe i am wrong? Also, does it mean no go for gmirror on ssd? No. It's fine. All SSDs in the past 15-20 years have wear leveling (and nearly all for an additional 10 years before that). It's quite hard to wear out a device by repeated writing to one sector. You effectively have to write the same amount of data you would if you were writing to multiple sectors. SSDs are rated in 'drive writes per day': how many times you can write to all the sectors of a drive, every day, for the warranty period of the device. This is between 0.3 and 5 typically (though exceptions exist). Any extra writes will be several orders of magnitude below this threshold for all but the most insane write patterns (eg write all the odd sectors, randomly, then write all the even sectors randomly, repeatedly). And if you are doing an insane amount of writing, you likely wouldn't be using gmirror.... It at most doubles the traffic to the drive, but if you have a 64k block size to UFS, you'd typically see only a few percent increase. So unless you are writing your data to the drives at rates approaching the endurance limit of the drive, this extra write won't be an issue.[*] Warner [*] It would theoretically be helpful,though, if gmirror could add an extra N sectors to match the underlying physical hardware page sizes, but the experiments I've done I've not been able to see a speed increase.... --0000000000009df31c05ef8ec4c6 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Sun, Dec 11, 2022 at 1:45 AM Artem= Kuchin <artemkuchin76@gmail.= com> wrote:
=20 =20 =20
11.12.2022 11:22, Warner Losh =D0=BF=D0=B8=D1=88=D0=B5=D1=82:
=20


On Sat, Dec 10, 2022, 11:52 PM Artem Kuchin <artemkuchin76@gmail.com> wrote:
Hello!

I am writing a small utility for myseld and part of it is comparing
gmirror disks. After running some tests i realized that some bytes at
the very end of disks are constantly changing.

The last sector has metadata about the mirror and about the mirror element.=C2=A0 It's this latter data tha= t differs.


Thank you for reply. Then there are several question

1) Last SECTOR is not always 512KB or is it? Do i need to get block size from diskinfo and subtract its size from disk size?


diskinfo(1) will tell you, it's ret= urned with the=C2=A0DIOCGSECTORSIZE ioctl.
=C2=A0

2) Why its content=C2=A0 is changing so often? On every write? How o= ften? The only place to look for description is the gmirror sources?

When a mirror breaks (that is, writes can happen to on= e side but not the other), we need to know right away which side is the mor= e current one. The gmirror does this by modifying the metadata to record ho= w many writes have happened to each mirror member (one reason that write is= so expensive).

> It does not look good to me, = but maybe i am wrong? Also, does it mean no go for gmirror on ssd?

<= div>No. It's fine. All SSDs in the past 15-20 years have wear leveling = (and nearly all for an additional 10 years before that). It's quite har= d to wear out a device by repeated writing to one sector. You effectively h= ave to write the same amount of data you would if you were writing to multi= ple sectors. SSDs are rated in 'drive writes per day': how many tim= es you can write to all the sectors of a drive, every day, for the warranty= period of the device. This is between 0.3 and 5 typically (though exceptio= ns exist).=C2=A0 Any extra writes will be several orders of magnitude=C2=A0= below this threshold for all but the most insane write patterns (eg write a= ll the odd sectors, randomly, then write all the even sectors randomly, rep= eatedly). And if you are doing an insane amount of writing, you likely woul= dn't be using gmirror.... It at most doubles the traffic to the drive, = but if you have a 64k block size to UFS, you'd typically see only a few= percent increase. So unless you are writing your data to the drives at rat= es approaching the endurance limit of the drive, this extra write won't= be an issue.[*]

Warner

[= *] It would theoretically be helpful,though, if gmirror could add an extra = N sectors to match the underlying physical hardware page sizes, but the exp= eriments I've done I've not been able to see a speed increase....
--0000000000009df31c05ef8ec4c6--