From nobody Mon Nov 20 20:25:02 2023 X-Original-To: freebsd-transport@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SYzXk2Dvsz51pM7 for ; Mon, 20 Nov 2023 20:25:42 +0000 (UTC) (envelope-from ccfreebsd@gmail.com) Received: from mail-lf1-f49.google.com (mail-lf1-f49.google.com [209.85.167.49]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SYzXk0L8yz3Vbv; Mon, 20 Nov 2023 20:25:42 +0000 (UTC) (envelope-from ccfreebsd@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lf1-f49.google.com with SMTP id 2adb3069b0e04-5094727fa67so6687911e87.3; Mon, 20 Nov 2023 12:25:41 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700511940; x=1701116740; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=EOdSvDrYugvu4mtBW+7dqqPNNUgI8BUyuV4+WeYWZ1Y=; b=Ad0rNhAKXyuU48Hc4LFtFeC/f5mfkSOeBIKHgJtwb0NKu/uLy8uXvkxv/lchpWmDvm mBR5GLElmSRTJPjfRiJfaeJIHL1HXRvAb9ZBvwgF/67+naHiFJslJRLavot45TKWXTl4 f9knGs4247PzR3L20VUeVr22gRFdeUMNN98rDU5PJhr8eGLCZQqJUxrgpLQlp2uMJDIj HKpITCVLO9OaekMQp3KlAomnNtXpAp3JclAfurIO518uNZU9Nh7BEC9ge8WlOJAyAsHf 2pKNZogk4HamsaBFJLEiZY40CJrLvh0ppRTvcL5oLf36lf+pGt2BZiqfHKhjzAox2SpY hjhA== X-Gm-Message-State: AOJu0YzYMstUIpBt4jOD45MemENf39tKkwMImFd2kEd2wRRTAcAq8vcM UFXVOdW0zxp7qs6jxvQaEWnv3UIz2F4CBA== X-Google-Smtp-Source: AGHT+IF07KEQENwUbKwvvER7NVdA3GnNCZHPp+g1gjLWVVMyWi+/AIyhIj3e14ns30xFt4NH8PFABw== X-Received: by 2002:ac2:55b8:0:b0:508:26b6:bc21 with SMTP id y24-20020ac255b8000000b0050826b6bc21mr6379740lfg.40.1700511939443; Mon, 20 Nov 2023 12:25:39 -0800 (PST) Received: from mail-lj1-f170.google.com (mail-lj1-f170.google.com. [209.85.208.170]) by smtp.gmail.com with ESMTPSA id e7-20020a196907000000b005092e621916sm1305749lfc.222.2023.11.20.12.25.39 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 20 Nov 2023 12:25:39 -0800 (PST) Received: by mail-lj1-f170.google.com with SMTP id 38308e7fff4ca-2c523ac38fbso61335011fa.0; Mon, 20 Nov 2023 12:25:39 -0800 (PST) X-Received: by 2002:a2e:8091:0:b0:2c5:14d1:a303 with SMTP id i17-20020a2e8091000000b002c514d1a303mr6015631ljg.25.1700511939147; Mon, 20 Nov 2023 12:25:39 -0800 (PST) List-Id: Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-transport List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-transport@freebsd.org X-BeenThere: freebsd-transport@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Cheng Cui Date: Mon, 20 Nov 2023 15:25:02 -0500 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Low performance of BBR compared to cubic To: "Scheffenegger, Richard" Cc: FreeBSD Transport Content-Type: multipart/alternative; boundary="00000000000011be27060a9b4a9d" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US] X-Rspamd-Queue-Id: 4SYzXk0L8yz3Vbv --00000000000011be27060a9b4a9d Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable > This is a quick test with iperf3 on bare metal ( old MBP i5 2 cores 4 > threads ). > The kernel current/15 with debug options disabled. Following is the > performance result: > > freebsd: 37.2 Gbits/sec 1.32 MBytes > RACK: 27.9 Gbits/sec 1.34 MBytes > BBR: 2.34 Gbits/sec 223 KBytes Without some detail of your testbed, I can not understand the result correctly. It's hard to understand why i5 (2 cores 4threads) was good enough to trigge= r such a large traffic near 40Gbps bandwidth. How many TCP flows are created in the test? I can only assume the traffic was not transferred on a wire. The cwnd diff of only 0.02 Mbytes could not explain the diff of near 10 Gbits/sec between freebsd and RACK. It made me think the TCP congestion control wasn'= t playing a role. Thus, the root cause might be somewhere else. Best Regards, Cheng Cui On Mon, Nov 20, 2023 at 9:11=E2=80=AFAM Scheffenegger, Richard wrote: > Zhenlei, > > Maybe you want to characterize the TX vs. RX codepaths independently of > each other, and run tests with different Stacks on either end (could be > tuned via setsockopt, or tcpsso; or maybe by togging the default in > between starting the iperf3 server side, before running the client, if > this is done via loopback). > > Currently, the expectation is that the TX codepath of RACK is more > optimized vs. the RX codepath - thus a RACK sender to a base stack > receiver should show the highest performance. > > Best reagrds, > Richard > > > Am 20.11.2023 um 13:29 schrieb Scheffenegger, Richard: > > > > > > BBR has not been looked after for quite some while (and there are no > > plans to invest there). > > > > Among other things, the blatant disregard of flow fairness which BBRv1 > > shows, makes it a poor protocol for general use. > > > > Similar issues also show up with BBRv2 (current version), but still it > > is not considered a reasonable and stable enough protocol - thus the > > IETF is working on BBRv3. > > > > Both of which would require effective re-implementation of the BBR stac= k. > > > > > > > > RACK is expected to perform better across congested paths, and in the > > presence of various pathological network issues. In particular, the > > receive path is certainly not as performant as the Base Stack currently= . > > > > In short: If you want to use BBR, please don't use the current code wit= h > > is at best a variation of BBRv1 - and it's generally known that this > > version of BBR is not "good". > > > > > > (I presume your testing was acrosss a green field / zero loss network, > > with ample bandwidth - maybe the loopback interface even). > > > > Richard > > > > > > > > -----Original Message----- > > > > > > Hi, > > > > While test TCP RACK functions, I tested BBR BTW. > > > > This is a quick test with iperf3 on bare metal ( old MBP i5 2 cores 4 > > threads ). > > The kernel current/15 with debug options disabled. Following is the > > performance result: > > > > freebsd: 37.2 Gbits/sec 1.32 MBytes > > RACK: 27.9 Gbits/sec 1.34 MBytes > > BBR: 2.34 Gbits/sec 223 KBytes > > > > For freebsd and RACK functions the CC is cubic. > > > > The last column is Cwnd. BBR's Cwnd looks quite small. > > > > There's also a report on Telegram Taiwan FreeBSD chat group, but withou= t > > performance details. > > > > I believe there is something wrong with BBR. This is not a reasonable > > good performance compared with other tcp congestion control algorithms. > > > > Or am I missing something ? > > > > Best regards, > > Zhenlei > > > > > --00000000000011be27060a9b4a9d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
> This is a quick test with iperf3 on = bare metal ( old MBP i5 2 cores 4
> threads ).
> The kernel current/15 with debug options disabled. Following is the > performance result:
>
> freebsd:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 37.2 Gbits/sec=C2= =A0 1.32 MBytes
> RACK:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 27.9= Gbits/sec=C2=A0 1.34 MBytes
> BBR:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= 2.34 Gbits/sec=C2=A0 223 KBytes

Without some detail of your testbed, I can not understand the resu= lt correctly.

It's hard to understand why i5 (= 2 cores 4threads) was good enough to trigger
such a large traffic= near 40Gbps bandwidth. How many TCP flows are created in
the tes= t? I can only assume the traffic was not transferred on a wire.
<= br>
The cwnd diff of only 0.02 Mbytes could not explain the diff = of near 10 Gbits/sec
between freebsd and RACK. It made me think t= he TCP congestion control wasn't
playing a role. Thus, the ro= ot cause might be somewhere else.

Best Regards,<= div>Cheng Cui


On Mon, Nov 20, 2023 at 9:11=E2=80= =AFAM Scheffenegger, Richard <rsc= heff@freebsd.org> wrote:
Zhenlei,

Maybe you want to characterize the TX vs. RX codepaths independently of each other, and run tests with different Stacks on either end (could be tuned via setsockopt, or tcpsso; or maybe by togging the default in
between starting the iperf3 server side, before running the client, if
this is done via loopback).

Currently, the expectation is that the TX codepath of RACK is more
optimized vs. the RX codepath - thus a RACK sender to a base stack
receiver should show the highest performance.

Best reagrds,
=C2=A0 =C2=A0Richard


Am 20.11.2023 um 13:29 schrieb Scheffenegger, Richard:
>
>
> BBR has not been looked after for quite some while (and there are no <= br> > plans to invest there).
>
> Among other things, the blatant disregard of flow fairness which BBRv1=
> shows, makes it a poor protocol for general use.
>
> Similar issues also show up with BBRv2 (current version), but still it=
> is not considered a reasonable and stable enough protocol - thus the <= br> > IETF is working on BBRv3.
>
> Both of which would require effective re-implementation of the BBR sta= ck.
>
>
>
> RACK is expected to perform better across congested paths, and in the =
> presence of various pathological network issues. In particular, the > receive path is certainly not as performant as the Base Stack currentl= y.
>
> In short: If you want to use BBR, please don't use the current cod= e with
> is at best a variation of BBRv1 - and it's generally known that th= is
> version of BBR is not "good".
>
>
> (I presume your testing was acrosss a green field / zero loss network,=
> with ample bandwidth - maybe the loopback interface even).
>
> Richard
>
>
>
> -----Original Message-----
>
>
> Hi,
>
> While test TCP RACK functions, I tested BBR BTW.
>
> This is a quick test with iperf3 on bare metal ( old MBP i5 2 cores 4 =
> threads ).
> The kernel current/15 with debug options disabled. Following is the > performance result:
>
> freebsd:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 37.2 Gbits/sec=C2= =A0 1.32 MBytes
> RACK:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 27.9= Gbits/sec=C2=A0 1.34 MBytes
> BBR:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= 2.34 Gbits/sec=C2=A0 223 KBytes
>
> For freebsd and RACK functions the CC is cubic.
>
> The last column is Cwnd. BBR's Cwnd looks quite small.
>
> There's also a report on Telegram Taiwan FreeBSD chat group, but w= ithout
> performance details.
>
> I believe there is something wrong with BBR. This is not a reasonable =
> good performance compared with other tcp congestion control algorithms= .
>
> Or am I missing something ?
>
> Best regards,
> Zhenlei
>
>
--00000000000011be27060a9b4a9d--