From nobody Tue May 02 10:04:33 2023 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q9bKJ2j7sz48n8T for ; Tue, 2 May 2023 10:04:36 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (turbocat.net [88.99.82.50]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q9bKH2p1Bz3Mtf for ; Tue, 2 May 2023 10:04:35 +0000 (UTC) (envelope-from hps@selasky.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of hps@selasky.org designates 88.99.82.50 as permitted sender) smtp.mailfrom=hps@selasky.org; dmarc=none Received: from [10.36.2.154] (unknown [46.212.121.255]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 21710262211; Tue, 2 May 2023 12:04:34 +0200 (CEST) Message-ID: <656f2daa-53a2-40d2-5fdc-b570473d56bc@selasky.org> Date: Tue, 2 May 2023 12:04:33 +0200 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.10.1 Subject: Re: Cwnd grows slowly during slow-start due to LRO of the receiver side. Content-Language: en-US From: Hans Petter Selasky To: Chen Shuo , freebsd-net@freebsd.org References: In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Result: default: False [-3.18 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-0.95)[-0.947]; NEURAL_HAM_SHORT(-0.93)[-0.931]; R_SPF_ALLOW(-0.20)[+a:mail.turbocat.net]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:88.99.0.0/16, country:DE]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; R_DKIM_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; DMARC_NA(0.00)[selasky.org]; TO_DN_SOME(0.00)[]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4Q9bKH2p1Bz3Mtf X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On 5/2/23 11:14, Hans Petter Selasky wrote: > Hi Chen! > > The FreeBSD mbufs carry the number of ACKs that have been joined > together into the following field: > > m->m_pkthdr.lro_nsegs > > Can this value be of any use to cc_newreno ? > > --HPS Hi Chen, Have you tested using FreeBSD main / 14 ? The "nsegs" are passed along like this: nsegs = max(1, m->m_pkthdr.lro_nsegs); ... cc_ack_received(tp, th, nsegs, CC_ACK); ... (Newreno - FreeBSD-14) incr = min(ccv->bytes_this_ack, ccv->nsegs * abc_val * CCV(ccv, t_maxseg)); And in FreeBSD-10 being mentioned in your article: (Newreno - FreeBSD-10) incr = min(ccv->bytes_this_ack, V_tcp_abc_l_var * CCV(ccv, t_maxseg)); There is no such thing. This issue may already have been fixed! --HPS > > On 5/2/23 09:46, Chen Shuo wrote: >> As per newreno_ack_received() in sys/netinet/cc/cc_newreno.c, >> FreeBSD TCP sender strictly follows RFC 5681 with RFC 3465 extension >> That is, during slow-start, when receiving an ACK of 'bytes_acked' >> >>      cwnd += min(bytes_acked, abc_l_var * SMSS);  // abc_l_var = 2 dflt >> >> As discussed in sec3.2 of RFC 3465, L=2*SMSS bytes exactly balances >> the negative impact of the delayed ACK algorithm.  RFC 5681 also >> requires that a receiver SHOULD generate an ACK for at least every >> second full-sized segment, so bytes_acked per ACK is at most 2 * SMSS. >> If both sender and receiver follow it. cwnd should grow exponentially >> during slow-slow: >> >>      cwnd *= 2    (per RTT) >> >> However, LRO and TSO are widely used today, so receiver may generate >> much less ACKs than it used to do.  As I observed, Both FreeBSD and >> Linux generates at most one ACK per segment assembled by LRO/GRO. >> The worst case is one ACK per 45 MSS, as 45 * 1448 = 65160 < 65535. >> >> Sending 1MB over a link of 100ms delay from FreeBSD 13.2: >> >>   0.000 IP sender > sink: Flags [S], seq 205083268, win 65535, options >> [mss 1460,nop,wscale 10,sackOK,TS val 495212525 ecr 0], length 0 >>   0.100 IP sink > sender: Flags [S.], seq 708257395, ack 205083269, win >> 65160, options [mss 1460,sackOK,TS val 563185696 ecr >> 495212525,nop,wscale 7], length 0 >>   0.100 IP sender > sink: Flags [.], ack 1, win 65, options [nop,nop,TS >> val 495212626 ecr 563185696], length 0 >>   // TSopt omitted below for brevity. >> >>   // cwnd = 10 * MSS, sent 10 * MSS >>   0.101 IP sender > sink: Flags [.], seq 1:14481, ack 1, win 65, >> length 14480 >> >>   // got one ACK for 10 * MSS, cwnd += 2 * MSS, sent 12 * MSS >>   0.201 IP sink > sender: Flags [.], ack 14481, win 427, length 0 >>   0.201 IP sender > sink: Flags [.], seq 14481:31857, ack 1, win 65, >> length 17376 >> >>   // got ACK of 12*MSS above, cwnd += 2 * MSS, sent 14 * MSS >>   0.301 IP sink > sender: Flags [.], ack 31857, win 411, length 0 >>   0.301 IP sender > sink: Flags [.], seq 31857:52129, ack 1, win 65, >> length 20272 >> >>   // got ACK of 14*MSS above, cwnd += 2 * MSS, sent 16 * MSS >>   0.402 IP sink > sender: Flags [.], ack 52129, win 395, length 0 >>   0.402 IP sender > sink: Flags [P.], seq 52129:73629, ack 1, win 65, >> length 21500 >>   0.402 IP sender > sink: Flags [.], seq 73629:75077, ack 1, win 65, >> length 1448 >> >> As a consequence, instead of growing exponentially, cwnd grows >> more-or-less quadratically during slow-start, unless abc_l_var is >> set to a sufficiently large value. >> >> NewReno took more than 20 seconds to ramp up throughput to 100Mbps >> over an emulated 100ms delay link.  While Linux took ~2 seconds. >> I can provide the pcap file if anyone is interested. >> >> Switching to CUBIC won't help, because it uses the logic in NewReno >> ack_received() for slow start. >> >> Is this a well-known issue and abc_l_var is the only cure for it? >> https://calomel.org/freebsd_network_tuning.html >> >> Thank you! >> >> Best, >> Shuo Chen >> > >