From nobody Fri Feb 18 08:36:18 2022 X-Original-To: freebsd-geom@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 5A97E19DFD92; Fri, 18 Feb 2022 08:36:24 +0000 (UTC) (envelope-from peterj@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4K0Q5h23wzz3krc; Fri, 18 Feb 2022 08:36:24 +0000 (UTC) (envelope-from peterj@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1645173384; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=Mi3SNNA0tJ81NnE97oPCcZrkn1pwqsn3FXY6VLeV9nk=; b=NuVQY8gj4J32CaN4omvBFfZ7Tz/yEsfXO1fphYyFsEP/rJvCgKeUgoLqHe0mTliYCtvlu/ 8Bcp+1H0qgy/Se9j7xELAlucwKZ/n1IhnM+77+1JSG9I704BnFFTeABKefQF0C405yJT5W fnf/oKi6uS4gibKHwoxTgUXT2ZAuRmfJR6S2QxzL4qp0irRqoB/zFmWI6/54qo7OhF67kb Me0Phja/FJRviW2wz46zeOKSyeR5PhTs44rpcj9tTJOk8rgnLHQE9Jwn+kO9fDTXoQnfwo AUa/6jRqXNLNCRWPXVig4GkNKVJ3KllZxFrohkh+cfzMVulKG+JVYFrQgk6hLw== Received: from server.rulingia.com (ppp239-208.static.internode.on.net [59.167.239.208]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: peterj) by smtp.freebsd.org (Postfix) with ESMTPSA id 305E62CC64; Fri, 18 Feb 2022 08:36:22 +0000 (UTC) (envelope-from peterj@freebsd.org) Date: Fri, 18 Feb 2022 19:36:18 +1100 From: Peter Jeremy To: FreeBSD FS , "freebsd-geom@FreeBSD.org" Subject: Re: bio re-ordering Message-ID: References: <9848cde6-5c12-cdd4-e722-42fe26fa0349@FreeBSD.org> <20220218014814.GJ97875@funkthat.com> List-Id: GEOM-specific discussions and implementations List-Archive: https://lists.freebsd.org/archives/freebsd-geom List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-geom@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="q22ohTV72yYk0JEy" Content-Disposition: inline In-Reply-To: <20220218014814.GJ97875@funkthat.com> X-PGP-Key: http://www.rulingia.com/keys/peter.pgp ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1645173384; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=Mi3SNNA0tJ81NnE97oPCcZrkn1pwqsn3FXY6VLeV9nk=; b=HkdFhopweco12sWSZJubr4/sSnL0VqZHI0AVJjJ/tGPjhBztJr44nPkK6FfN/KJ2T9a7an qXB8oPiNL2PKLxdT6O7mSw/Wz2Ll0KMA4vaag9bO8AaMwxQwBsN09cFH5LUVZQAiGSget+ eynS2x/G5LgiwHRDiHET1hn+dHsk4etmm67GOVNnRf96cNlnh8xPseTGDdWBmzpfxQ7t47 3u5xwLfqWKxZS2PUrBIAspc9nQSm+/aze0mudPXHP3CyoqebNuHqYzTH77la8MsUPUsfmG TX+730wH3vb1WTzKsLL47cxODFFWmNYuvxuk6tVHORZ8d+oihbsVbndZScpbKw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1645173384; a=rsa-sha256; cv=none; b=Jh324HppuCQzLywXKfy3uIp0nq49F4uJzr+vMB/kvgbwusxNvqXLGkqsJyOeJdF87yVLqG KRc5AyLz9mEoW1hgVg/BrnVtQ5pPxXh/Sd9R6XqSHecmmDgg26ppUJV/YdPLGiYD/9LmST wNLaCy52XoVdsJO29XuTGz3jZeXnQGYgtedChmEdmgaiL06YVwJh/jsPrJ+t/KBLCY+e71 hs2QZQZJZqGMN4cilVXLzq+E8ZhDELLT7+ksXS3HsxHy8wJS68KIqqai6bN9CsWIMKLfhP kZ5htmzNMYyQB99FgknHK+q1tDoZKa5jdooz+azAJdvBNWdKdeINbdpsMEeraA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N --q22ohTV72yYk0JEy Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2022-Feb-17 17:48:14 -0800, John-Mark Gurney wrote: >Peter Jeremy wrote this message on Sat, Feb 05, 2022 at 20:50 +1100: >> I've raised https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D261731 to >> make geom_gate support BIO_ORDERED. Exposing the BIO_ORDERED flag to >> userland is quite easy (once a decision is made as to how to do that). >> Enhancing the geom_gate clients to correctly implement BIO_ORDERED is >> somewhat harder. > >The clients are single threaded wrt IOs, so I don't think updating them >are required. ggatec(8) and ggated(8) will not reorder I/Os. I'm not sure about hast. >I do have patches to improve things by making ggated multithreaded to >improve IOPs, and so making this improvement would allow those patches >to be useful. Likewise, I found ggatec and ggated to be too slow for my purposes and so I've implemented my own variant (not network API compatible) that can/does reorder requests. That was when I noticed that BIO_ORDERED wasn't implemented. >I do have a question though, what is the exact semantics of _ORDERED? I can't authoritatively answer this, sorry. >And right now, the ggate protocol (from what I remember) doesn't have >a way to know when the remote kernel has received notification that an >IO is complete. A G_GATE_CMD_START write request will be sent to the remote system and issued as a pwrite(2) then an acknowledgement packet will be returned and passed back to the local kernel via G_GATE_CMD_DONE. There's no support for BIO_FLUSH or BIO_ORDERED so there's no way for the local kernel to know when the write has been written to non-volatile store. >> I've done some experiments and OpenZFS doesn't generate BIO_ORDERED >> operations so I've also raised https://github.com/openzfs/zfs/issues/130= 65 >> I haven't looked into how difficult that would be to fix. Unrelated to the above but for completeness: OpenZFS avoids the need for BIO_ORDERED by not issuing additional I/Os until previous I/Os have been retired when ordering is important. (It does rely on BIO_FLUSH). --=20 Peter Jeremy --q22ohTV72yYk0JEy Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEE7rKYbDBnHnTmXCJ+FqWXoOSiCzQFAmIPWn1fFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEVF QjI5ODZDMzA2NzFFNzRFNjVDMjI3RTE2QTU5N0EwRTRBMjBCMzQACgkQFqWXoOSi CzQoug/9HBdIH8xPRej9/4/cO0136I3bxqIynTIvQiIvDcdy9E1DbqrfSV5yB7dR GyvVaIFMCvR3PljmDpGU9J7Gr/uzohcOOX3oU7mZ0XENGYFztbZrQ7ZAsw9KGgPP WgTNqrXlGkMDx1RALf/KRU/YhiRg7alQAl/ulTfwxyG9OixzDQT5JcD4tOv3SIzR ouhLo3MaXvy0NbNERAzelLNJApE+P5SM1i7XUFEY2ri5KmSkfV2ogV+E4ZZCu5Jj 6pb5u2gFEQDMeH+tfDxSGdjqV3ssP5hwY2wjzweMEpS1cJ2hvE/oLMAo9+/Z+mQN PtlYxYqIWtV6R+ZArbcyHUWXctvSO0lRTcwe62PqEAcNANLUl4RNrK3RZ12xsNwT HRyZE8LOK1dTHratF02UJ6dLaajR4CXRFkiJu6xfN8Hy8R9oejTcezICiWEYsZV1 l9QHKW2Vr8xwJjVjAz+YZBvWRoD7MQS8Am4/6zMfBYvosJU9KGHXMKroi0v43Hl0 UDC0yQtaHhIE4qPM64UtonkFKR1bC8fJJYVeO+hj9s1rA1dZG+c+tIuLbVPVQ8Hg 8fk18jy7IeksOHM9maO54kf5Oj77alrs1XjpX6wEW+0/pRFula4OluV+fDoZTv7J uoa0fVm+uEDlRd3bzOitVdXh3vMvMmiF/4yCLhiTET58e6CQTjc= =RNi/ -----END PGP SIGNATURE----- --q22ohTV72yYk0JEy--