From nobody Fri Feb 18 16:31:13 2022 X-Original-To: freebsd-geom@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 59E3219CE123 for ; Fri, 18 Feb 2022 16:31:33 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ua1-x936.google.com (mail-ua1-x936.google.com [IPv6:2607:f8b0:4864:20::936]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4K0cdw3YQbz3qfQ for ; Fri, 18 Feb 2022 16:31:32 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-ua1-x936.google.com with SMTP id g18so4534523uak.5 for ; Fri, 18 Feb 2022 08:31:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20210112.gappssmtp.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=DfM7FmnDyvj93Hq+rlHbfv7sKthuIm5IHBzHrT4W0X0=; b=aZiMq2WJWbQO4PiGsD7cYVQTbdTI8dOjF4/BLS7SJqpCfmkbZ/wNana3jcEB2v7BhJ kmZ/5N7/Puea65hyCy8VQK8YTyLMPD0+7Jcz0QjgPowoz2OZC6pl/Oc8robxCs6wtpwL ArPhfIbU6ijb6wxnaJYOGUVLNLwGfNXxcnPuZUj+MWLkOzMoL9Wacg9jbv8QlGmWyqOB +bUVAjn7vXGuCpufcNQjsAGjUSM5eiZmvnLm21Kd/obqQAOf4TTg3dK2tsK77x9fOMsh fTgRLqTfimUZRkO3nsw7Sr770nI4SSHMaThVWD9aE9wCD6+pcBK9Iy1Dts9TBmVNe8Lk OUhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=DfM7FmnDyvj93Hq+rlHbfv7sKthuIm5IHBzHrT4W0X0=; b=Qv5OnaemXl5zsP21peFJqcGi/5MHbXJUUy2CccaMQ8NqRn4t9Wq75mlbmorydYLKT+ 9dzXBcFc+MtBeU+oUo/1I1Pv93IwBNNwoNuuLEAEbVHXp7HjFhrsuW+qYbyM8p0dytHU LtoPgwIYAqvmjoI/HkM+7llqVjHs9IejPx5LiIo/2gN59F75fBrBSlqTn1fLUu5u6Vrr 61Y47FeMBu07HzTn8YP+e4u3zM3gd5Yvgu8XnBCxL/p1ueJ9+ljHf8ku3FDyxFbjmJrR wD5V6vxvK+jG/03yNFPs3tWH0I2xAkvQO9p1LLO9juK2b2AAYtpPiJJKDbRtoR2NTWxF K1NA== X-Gm-Message-State: AOAM531mQNuUZSdJoWIPqf8BJSqL/leFxqau3z2+izawHvMDPVv3zNIQ PkdF25EqT86W6kIgszHybEZOxmXSkCSs3URBfwQIMA== X-Google-Smtp-Source: ABdhPJztBpdSw7moYd7bY191fRxYYL1rvkRZTpRl9cZUrHzwc0KMTNjBr34k2DaCOHk+bYCsUUd5twk3sv7ipaL+7ww= X-Received: by 2002:ab0:1530:0:b0:33c:53cf:6844 with SMTP id o45-20020ab01530000000b0033c53cf6844mr3266746uae.85.1645201885697; Fri, 18 Feb 2022 08:31:25 -0800 (PST) List-Id: GEOM-specific discussions and implementations List-Archive: https://lists.freebsd.org/archives/freebsd-geom List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-geom@freebsd.org MIME-Version: 1.0 References: <9848cde6-5c12-cdd4-e722-42fe26fa0349@FreeBSD.org> <20220218014814.GJ97875@funkthat.com> In-Reply-To: From: Warner Losh Date: Fri, 18 Feb 2022 09:31:13 -0700 Message-ID: Subject: Re: bio re-ordering To: Peter Jeremy Cc: FreeBSD FS , "freebsd-geom@FreeBSD.org" Content-Type: multipart/alternative; boundary="000000000000fb45dd05d84d69e2" X-Rspamd-Queue-Id: 4K0cdw3YQbz3qfQ X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20210112.gappssmtp.com header.s=20210112 header.b=aZiMq2WJ; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::936) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-2.89 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20210112.gappssmtp.com:s=20210112]; NEURAL_HAM_MEDIUM(-1.00)[-0.996]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-0.90)[-0.905]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-geom@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20210112.gappssmtp.com:+]; NEURAL_HAM_SHORT(-0.99)[-0.993]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::936:from]; MLMMJ_DEST(0.00)[freebsd-geom]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; R_SPF_NA(0.00)[no SPF record]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N --000000000000fb45dd05d84d69e2 Content-Type: text/plain; charset="UTF-8" So I spent some time looking at what BIO_ORDERED means in today's kernel and flavored it with my indoctrination of the ordering guarantees with BIO requests from when I wrote the CAM I/O scheduler. it's kinda long, but spells out what BIO_ORDERED means, where it can come from and who depends on it for what. On Fri, Feb 18, 2022 at 1:36 AM Peter Jeremy wrote: > On 2022-Feb-17 17:48:14 -0800, John-Mark Gurney wrote: > >Peter Jeremy wrote this message on Sat, Feb 05, 2022 at 20:50 +1100: > >> I've raised https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=261731 to > >> make geom_gate support BIO_ORDERED. Exposing the BIO_ORDERED flag to > >> userland is quite easy (once a decision is made as to how to do that). > >> Enhancing the geom_gate clients to correctly implement BIO_ORDERED is > >> somewhat harder. > > > >The clients are single threaded wrt IOs, so I don't think updating them > >are required. > > ggatec(8) and ggated(8) will not reorder I/Os. I'm not sure about hast. > > >I do have patches to improve things by making ggated multithreaded to > >improve IOPs, and so making this improvement would allow those patches > >to be useful. > > Likewise, I found ggatec and ggated to be too slow for my purposes and > so I've implemented my own variant (not network API compatible) that > can/does reorder requests. That was when I noticed that BIO_ORDERED > wasn't implemented. > > >I do have a question though, what is the exact semantics of _ORDERED? > > I can't authoritatively answer this, sorry. > This is under documented. Clients, in general, are expected to cope with I/O that completes in an arbitrary order. They are expected to not schedule new I/O that depends on old I/O completing for whatever reason (usually on-media consistency). BIO_ORDERED is used to create a full barrier in the stream of I/Os. The comments in the code say vaguely: /* * This bio must be executed after all previous bios in the queue have been * executed, and before any successive bios can be executed. */ Drivers implement this as a partitioning of requests. All requests before it are completed, then the BIO_ORDERED operation is done, then requests after it are scheduled with the device. BIO_FLUSH I think is the only remaining operation that's done as BIO_ORDERED directly. xen.../blkback.c, geom_io.c and ffs_softdep.c are the only ones that set it and all on BIO_FLUSH operations. bio/buf clients depend on this to ensure metadata on the drive is in a consistent state after it's been updated. xen/.../blkback.c also sets it for all BLKIF_OP_WRITE_BARRIER operations (so write barriers). In the upper layers, we have struct buf instead of struct bio to describe future I/Os that the buffer cache may need to do. There's a flag B_BARRIER that gets turned into BIO_ORDERED in geom_vfs. B_BARRIER is set in only two places (and copied in one other) in vfs_bio.c. babarrierwrite and bbarrierwrite for async vs sync writes respectively. CAM will set BIO_ORDERED for all BIO_ZONE commands for reasons that are at best unclear to me, but which won't matter for this discussion. ffs_alloc.c (so UFS again) is the only place that uses babarrierwrite. It is used to ensure that all inode initializations are completed before the cylinder group bitmap is written out. This is done with newfs, when new cylinder groups are created with growfs, and apparently in a few other cases where additional inodes are created in newly-created UFS2 filesystems. This can be disabled with vfs.ffs.doasyncinodeinit=0 when barrier writes aren't working as advertised, but there's a big performance hit from doing so until all the inodes for the filesystem have been lazily populated. No place uses bbarrierwrite that I can find. Based on all of that, the CAM's dynamic I/O scheduler will reorder reads around a BIO_ORDERED operation, but not writes, trims or flushes. Since, in general, operations happen in an arbitrary order, scheduling both a read and a write at the same time for the same block will result in undefined results. Different drivers handle this differently. CAM will honor the BIO_ORDERED tag by scheduling the I/O with an ordering tag so that the SCSI hardware will properly order the result. The simpler ATA version will use a non NCQ request to force the proper ordering (since to send a non-NCQ request, you have to drain the queue, do that one command, and then start up again). nvd will just throw the I/O at the device, until it encounters a BIO_ORDERED request. Then it will queue everything until all the current requests complete, then do the ordered request, then do the rest of the queued I/O as if it had just showed up. Most drivers use bioq_disksort(), which will queue the request to the end of the bioq and mark things so all I/Os after that are in their new 'elevator car' for its elevator sort algorithm. This means that CAM's normal ways of dequeuing the request will preserve ordering through the periph driver's start routine (where the dynamic schedule will honor it for writes, but not reads, but the default scheduler will honor it for both). > >And right now, the ggate protocol (from what I remember) doesn't have > >a way to know when the remote kernel has received notification that an > >IO is complete. > > A G_GATE_CMD_START write request will be sent to the remote system and > issued as a pwrite(2) then an acknowledgement packet will be returned > and passed back to the local kernel via G_GATE_CMD_DONE. There's no > support for BIO_FLUSH or BIO_ORDERED so there's no way for the local > kernel to know when the write has been written to non-volatile store. > That's unfortunate. UFS can work around the BIO_ORDERED problem with a simple setting, but not the BIO_FLUSH problem. > >> I've done some experiments and OpenZFS doesn't generate BIO_ORDERED > >> operations so I've also raised > https://github.com/openzfs/zfs/issues/13065 > >> I haven't looked into how difficult that would be to fix. > > Unrelated to the above but for completeness: OpenZFS avoids the need > for BIO_ORDERED by not issuing additional I/Os until previous I/Os have > been retired when ordering is important. (It does rely on BIO_FLUSH). > To be clear: OpenZFS won't schedule new I/Os until the BIO_FLUSH it sends down w/o the BIO_ORDERED flag completes, right? The parenthetical confuses me on how to parse it: BIO_FLUSH is needed and ZFS depends on it completing with all blocks flushed to stable media, or ZFS depends on BIO_FLUSH being strongly ordered relative to other commands. I think you mean the former, but want to make sure. The root of this problem, I think, is the following: % man 9 bio No manual entry for bio I think I'll have to massage this email into an appropriate man page. At the very least, I should turn some/all of the above into a blog post :) Warner --000000000000fb45dd05d84d69e2 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
So I spent some time looking at what BIO_ORDERED mean= s in today's kernel
and flavored it with my indoctrination of= the ordering guarantees with BIO requests
from when I wrote the = CAM I/O scheduler. it's kinda long, but spells out what
BIO_O= RDERED means, where it can come from and who depends on it for what.
<= br>
On Fri,= Feb 18, 2022 at 1:36 AM Peter Jeremy <peterj@freebsd.org> wrote:
On 2022-Feb-17 17:48:14 -0800, John-Mark Gurney <= jmg@funkthat.com&= gt; wrote:
>Peter Jeremy wrote this message on Sat, Feb 05, 2022 at 20:50 +1100: >> I've raised https://bugs.free= bsd.org/bugzilla/show_bug.cgi?id=3D261731 to
>> make geom_gate support BIO_ORDERED.=C2=A0 Exposing the BIO_ORDERED= flag to
>> userland is quite easy (once a decision is made as to how to do th= at).
>> Enhancing the geom_gate clients to correctly implement BIO_ORDERED= is
>> somewhat harder.
>
>The clients are single threaded wrt IOs, so I don't think updating = them
>are required.

ggatec(8) and ggated(8) will not reorder I/Os.=C2=A0 I'm not sure about= hast.

>I do have patches to improve things by making ggated multithreaded to >improve IOPs, and so making this improvement would allow those patches<= br> >to be useful.

Likewise, I found ggatec and ggated to be too slow for my purposes and
so I've implemented my own variant (not network API compatible) that can/does reorder requests.=C2=A0 That was when I noticed that BIO_ORDERED wasn't implemented.

>I do have a question though, what is the exact semantics of _ORDERED?
I can't authoritatively answer this, sorry.

This is under documented. Clients, in general, are expected to cop= e with
I/O that completes in an arbitrary order. They are expecte= d to not schedule
new I/O that depends on old I/O completing for = whatever reason (usually
on-media consistency). BIO_ORDERED is us= ed to create a full barrier
in the stream of I/Os. The comments i= n the code say vaguely:

/*
=C2=A0* This bio mus= t be executed after all previous bios in the queue have been
=C2=A0* exe= cuted, and before any successive bios can be executed.
=C2=A0*/

Drivers implement this as a partitioning of requests. = All requests before
it are completed, then the BIO_ORDERED operat= ion is done, then requests
after it are scheduled with the device= .

BIO_FLUSH I think is the only remaining operatio= n that's done as BIO_ORDERED
directly. xen.../blkback.c, geom= _io.c and ffs_softdep.c are the only ones that set it
and all on = BIO_FLUSH operations. bio/buf clients depend on this to ensure metadata
on the drive is in a consistent state after it's been updated.

xen/.../blkback.c also sets it for all=C2=A0BLKIF_O= P_WRITE_BARRIER operations (so
write barriers).

In the upper layers, we have struct buf instead of struct bio to de= scribe future I/Os
that the buffer cache may need to do. Ther= e's a flag B_BARRIER that gets turned
into BIO_ORDERED in geo= m_vfs. B_BARRIER is set in only two places (and copied
in one oth= er) in vfs_bio.c.=C2=A0babarrierwrite and=C2=A0bbarrierwrite for async vs s= ync writes
respectively.

CAM will set BI= O_ORDERED for all BIO_ZONE commands for reasons that are
at best = unclear to me, but which won't matter for this discussion.
ffs_alloc.c (so UFS again) is the only place that uses=C2=A0ba= barrierwrite. It is used
to ensure that all inode initializations= are completed before the cylinder group
bitmap is written out. T= his is done with newfs, when new cylinder groups are
created with= growfs, and apparently in a few other cases where additional inodes
<= div>are created in newly-created UFS2 filesystems. This can be disabled wit= h
vfs.ffs.doasyncinodeinit=3D0 when barrier writes aren't wor= king as advertised,
but there's a big performance hit from do= ing so until all the inodes for the
filesystem have been lazily p= opulated.

No place uses bbarrierwrite that I can f= ind.

Based on all of that, the CAM's dynamic I= /O scheduler will reorder reads
around a BIO_ORDERED operation, b= ut not writes, trims or flushes. Since,
in general, operations ha= ppen in an arbitrary order, scheduling both a read
and a write at= the same time for the same block will result in undefined
result= s.

Different drivers handle this differently. CAM = will honor the BIO_ORDERED
tag by scheduling the I/O with an orde= ring tag so that the SCSI hardware will
properly order the result= . The simpler ATA version will use a non NCQ request
to force the= proper ordering (since to send a non-NCQ request, you have to
dr= ain the queue, do that one command, and then start up again). nvd will just= throw
the I/O at the device, until it encounters a BIO_ORDERED r= equest. Then it will queue
everything until all the current reque= sts complete, then do the ordered request, then
do the rest of th= e queued I/O as if it had just showed up.

Most dri= vers use bioq_disksort(), which will queue the request to the end of the bi= oq
and mark things so all I/Os after that are in their new 'e= levator car' for its elevator sort
algorithm. This means that= CAM's normal ways of dequeuing the request will preserve
ord= ering through the periph driver's start routine (where the dynamic sche= dule will honor
it for writes, but not reads, but the default sch= eduler will honor it for both).
=C2=A0
>And right now, the ggate protocol (from what I remember) doesn't ha= ve
>a way to know when the remote kernel has received notification that an<= br> >IO is complete.

A G_GATE_CMD_START write request will be sent to the remote system and
issued as a pwrite(2) then an acknowledgement packet will be returned
and passed back to the local kernel via G_GATE_CMD_DONE.=C2=A0 There's = no
support for BIO_FLUSH or BIO_ORDERED so there's no way for the local kernel to know when the write has been written to non-volatile store.

That's unfortunate. UFS can work around = the BIO_ORDERED problem with
a simple setting, but not the BIO_FL= USH problem.
=C2=A0
>> I've done some experiments and OpenZFS doesn't generate BI= O_ORDERED
>> operations so I've also raised https://githu= b.com/openzfs/zfs/issues/13065
>> I haven't looked into how difficult that would be to fix.

Unrelated to the above but for completeness:=C2=A0 OpenZFS avoids the need<= br> for BIO_ORDERED by not issuing additional I/Os until previous I/Os have
been retired when ordering is important.=C2=A0 (It does rely on BIO_FLUSH).=

To be clear: OpenZFS won't schedul= e new I/Os until the BIO_FLUSH it sends
down w/o the BIO_ORDERED = flag completes, right? The parenthetical confuses
me on how to pa= rse it: BIO_FLUSH is needed and ZFS depends on it completing
with= all blocks flushed to stable media, or ZFS depends on BIO_FLUSH being
strongly ordered relative to other commands. I think you mean the for= mer, but want
to make sure.

The root of = this problem, I think, is the following:
=C2=A0 =C2=A0 =C2=A0% ma= n 9 bio
=C2=A0 =C2=A0 =C2=A0No manual entry for bio
I think I'll have to massage this email into an appropria= te man page.

At the very least, I should turn some/all of the above into a bl= og post :)

Warner
=
--000000000000fb45dd05d84d69e2--