From nobody Wed Feb 02 09:49:44 2022 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 79ACD19A8AB0; Wed, 2 Feb 2022 09:49:48 +0000 (UTC) (envelope-from avg@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JpcTm300bz3q9p; Wed, 2 Feb 2022 09:49:48 +0000 (UTC) (envelope-from avg@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1643795388; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FA/jePrRhJzXTIWIr4FdLmyyvsEjugB9sd9EXmYpY9s=; b=N5wnpuAZItff9/8eh0SSFc/WPQE8cexWpY/JXRQZ9uDQA0bPhm8aNezc5u7MCnOtHb9yiz 15Ox4lnwAcaLjU8NanYdZYmb5Ob4XE9srxU9KqysWiFA7eCj2yGfjAdaPynZgscVGMi3bT cpo4p1x6zLQ3D8qFm0b7lEfO/s+clZlPeK49Wg2EgA5qzzOD3nEkKC/Ect52fDE46WSFmp 19ta3scGuI4NkzdKwnEEAIlacXJGyOW7hvZa0gdQy3F+kQ+8J+ZOFIjLfxakTdO/khGk6A +omk0JDdBCF+jwpHvX00AdzHoeUjJabQ2TdTtXftKV7k04XmFSoHNm/9dvd2PA== Received: from [192.168.0.88] (unknown [195.64.148.76]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: avg/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 89C4979E2; Wed, 2 Feb 2022 09:49:47 +0000 (UTC) (envelope-from avg@freebsd.org) Message-ID: <9848cde6-5c12-cdd4-e722-42fe26fa0349@FreeBSD.org> Date: Wed, 2 Feb 2022 11:49:44 +0200 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:91.0) Gecko/20100101 Firefox/91.0 Thunderbird/91.5.1 From: Andriy Gapon Subject: Re: bio re-ordering Content-Language: en-US To: Warner Losh Cc: Peter Jeremy , Konstantin Belousov , FreeBSD FS , "freebsd-geom@FreeBSD.org" References: In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1643795388; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FA/jePrRhJzXTIWIr4FdLmyyvsEjugB9sd9EXmYpY9s=; b=vmc8326HyVOWNHVQ/qp/GJnLa2h7UZVmtsup2gfTwSbJuTULJzfN54zB77Ae0wVUD/BR8a 3ZcPDQAbiyhkPvS7c4o8mYdDCTlF/Sg26DnFBEuWLI230efF7EweIsh+xq0Xd2j7AOMbB1 B68AXx4edzkPzer59d3/hSXs7KH7IvNU6XncQJkr+I3S9haocc7TbvLXPq/fSmSobyyeRA 04PoPPCdL1X+D5LIMuTfuS4+IM8GOjtTWJ1l7Vpvy7dYzdU71rATk3E5WuvN5dZ1o8MRtb rff0ZbqQ6kp3+ZJswrd9a1B2at9ISQ/dZz/96sMIBTyyW2rv5lFRerUBfZTV1g== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1643795388; a=rsa-sha256; cv=none; b=WW+t8UJMR9gdb6rdluD5IlGvyz2+yvfK78N5+2U8XHS1h97X8k7HjZY3CWUCaBSA8OqHf7 vSS3rzqbzrxAA1CAifyvzfVmKkBmNPd4eSSIKGgWaStPxLEfHSi/1gS9it1SK6d1W8Cikj evQlHf47+80TjiZIeLh2eCUfY1w3IJ14vfvCmpqqNtFGnsOFbItOJGm84n7BcrxzSSndX2 bY3lT0RGyLay51txEF69v7bLh8iQHbUINfTTfJz+K7RDhMo7zTQD0ptGBz/W1CjQceNXo3 YEUPYwllWO6fE4LHRNogM6jDlwQiq+ukdOstb5ZdOORdkJox9/6WzGB5pEaPCQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N On 02/02/2022 11:14, Warner Losh wrote: > On Wed, Feb 2, 2022 at 2:05 AM Andriy Gapon > wrote: > Hmm... it looks like both the old and new (Open)ZFS use BIO_FLUSH command > without BIO_ORDERED flag.  Not sure if it happens to do the right thing anyway > or not. > > > It's an unordered flush then. The flush will happen whenever. I have a vague > memory that ZFS will only issue this command in cases where there's no other I/O > pending. I think that there is still a potential problem that an earlier write request might get re-ordered after the flush. I think that we should add BIO_ORDERED for correctness. > It will be the only way for it to be reliable with nvme, since the > BIO_FLUSH > command isn't ordered w/o BIO_ORDERED flag. So ggate needn't do anything > special for BIO_FLUSH, just BIO_ORDERED. Otherwise, it's free to reorder as it > sees fit. > > The CAM I/O scheduler takes a little bit of liberty here, btw. It interprets > BIO_ORDERED > as being only wrt BIO_WRITE and BIO_FLUSH because if you schedule both a read > and write, the results are undefined. nvd takes a stricter approach and honors > the ordering > more strictly. > > Warner > -- Andriy Gapon