From nobody Tue Jan 31 21:32:20 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P5yv92LH9z3cMRd for ; Tue, 31 Jan 2023 21:32:37 +0000 (UTC) (envelope-from paul@gromit.dlib.vt.edu) Received: from gromit.dlib.vt.edu (gromit.dlib.vt.edu [128.173.126.123]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4P5yv86T4qz3jKT for ; Tue, 31 Jan 2023 21:32:36 +0000 (UTC) (envelope-from paul@gromit.dlib.vt.edu) Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of paul@gromit.dlib.vt.edu has no SPF policy when checking 128.173.126.123) smtp.mailfrom=paul@gromit.dlib.vt.edu; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=vt.edu (policy=none) Received: from smtpclient.apple (unknown [IPv6:2001:470:e15b:23::23]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by gromit.dlib.vt.edu (Postfix) with ESMTPSA id AD3705197A; Tue, 31 Jan 2023 16:32:31 -0500 (EST) From: Paul Mather Message-Id: <8A1B625C-D948-43C4-B787-DFB8A25BBF7A@gromit.dlib.vt.edu> Content-Type: multipart/alternative; boundary="Apple-Mail=_AE7EC00B-3170-4B90-9FE3-DF492002C429" List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Slow WAN traffic to FreeBSD hosts but not to Linux hosts---how to debug/fix? Date: Tue, 31 Jan 2023 16:32:20 -0500 In-Reply-To: Cc: freebsd-stable@freebsd.org To: Matt Garber References: <95EDCFCA-7E3F-458F-85A6-856D606B9D98@gromit.dlib.vt.edu> X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Spamd-Result: default: False [-1.49 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.997]; NEURAL_HAM_LONG(-0.99)[-0.994]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[vt.edu : No valid SPF, No valid DKIM,none]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:1312, ipnet:128.173.0.0/16, country:US]; RCPT_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_NA(0.00)[no SPF record]; FREEFALL_USER(0.00)[paul]; ARC_NA(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_DN_SOME(0.00)[]; TAGGED_RCPT(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4P5yv86T4qz3jKT X-Spamd-Bar: - X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_AE7EC00B-3170-4B90-9FE3-DF492002C429 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 On Jan 30, 2023, at 6:30 PM, Matt Garber wrote: >=20 >> > Any help/insight is gratefully appreciated. >> >=20 >> > Cheers, >> >=20 >> > Paul. >> >=20 >>=20 >> sysctl net.inet.tcp.cc.algorithm=3Dhtcp >>=20 >> I would set "htcp" on the server and home computer to improve through = in=20 >> your type of situation. >=20 >=20 > There may be other FreeBSD sysctls that have bad defaults in this = scenario and could be better tuned, but I doubt changing the CC = algorithm at this point is the problem =E2=80=94 at least not so much a = problem that=E2=80=99s causing throughput to be reduced so drastically. = Happy to be wrong if that does help things quickly and easily, though. Changing the CC algorithm doesn't affect speeds greatly in my case. For = example, changing to HTCP on one of the remote FreeBSD systems improves = speeds somewhat, but not to put them on a par with what I get from the = Linux systems. I can easily get 19--20 MB/s on a 100 MB download from = Linux but I've never managed to get above 10 MB/s at all from the remote = FreeBSD systems, even when switching to HTCP. In addition, the Linux = download quickly gets up to max speed but FreeBSD is slower to increase = speeds and is more variable in the speeds throughout the entire = download. > (Since OP mentioned that FreeBSD CC was set to CUBIC, that would match = what the Linux boxes are using by default, too, unless they=E2=80=99ve = been changed to something newer like BBR=E2=80=A6 so seems like CUBIC = *should* be performing fine on this WAN link, and the difference is = something else.) I suspect the same thing, too, i.e., it is implementation/specific = configuration details that differ between FreeBSD and Linux, especially = when they are both set to use CUBIC. I checked on one of the Linux systems (Rocky Linux 9) and verified it = defaults to CUBIC (net.ipv4.tcp_congestion_control =3D cubic). Cheers, Paul. --Apple-Mail=_AE7EC00B-3170-4B90-9FE3-DF492002C429 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 On Jan 30, = 2023, at 6:30 PM, Matt Garber <matt.garber@gmail.com> = wrote:


> Any help/insight is gratefully appreciated.
>
> Cheers,
>
> Paul.
>

sysctl net.inet.tcp.cc.algorithm=3Dhtcp

I would set "htcp" on the server and home computer to improve through in =
your type of situation.


There may = be other FreeBSD sysctls that have bad defaults in this scenario and = could be better tuned, but I doubt changing the CC algorithm at this = point is the problem =E2=80=94 at least not so much a problem that=E2=80=99= s causing throughput to be reduced so drastically. Happy to be wrong if = that does help things quickly and easily, = though.


=
Changing the CC algorithm doesn't affect speeds greatly in my case. =  For example, changing to HTCP on one of the remote FreeBSD systems = improves speeds somewhat, but not to put them on a par with what I get = from the Linux systems.  I can easily get 19--20 MB/s on a 100 MB = download from Linux but I've never managed to get above 10 MB/s at all = from the remote FreeBSD systems, even when switching to HTCP.  In = addition, the Linux download quickly gets up to max speed but FreeBSD is = slower to increase speeds and is more variable in the speeds throughout = the entire download.


(Since OP mentioned that = FreeBSD CC was set to CUBIC, that would match what the Linux boxes are = using by default, too, unless they=E2=80=99ve been changed to something = newer like BBR=E2=80=A6 so seems like CUBIC *should* be performing fine = on this WAN link, and the difference is something = else.)


I = suspect the same thing, too, i.e., it is implementation/specific = configuration details that differ between FreeBSD and Linux, especially = when they are both set to use CUBIC.

I checked = on one of the Linux systems (Rocky Linux 9) and verified it defaults to = CUBIC (net.ipv4.tcp_congestion_control =3D = cubic).

Cheers,

Paul.
= --Apple-Mail=_AE7EC00B-3170-4B90-9FE3-DF492002C429--