From nobody Tue Jan 31 21:12:48 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P5ySX2yz6z3cJwf for ; Tue, 31 Jan 2023 21:13:00 +0000 (UTC) (envelope-from paul@gromit.dlib.vt.edu) Received: from gromit.dlib.vt.edu (gromit.dlib.ipv6.vt.edu [IPv6:2001:468:c80:a103:2:5000:5555:5555]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4P5ySX1G4Nz3Ptj for ; Tue, 31 Jan 2023 21:13:00 +0000 (UTC) (envelope-from paul@gromit.dlib.vt.edu) Authentication-Results: mx1.freebsd.org; none Received: from smtpclient.apple (unknown [IPv6:2001:470:e15b:23::23]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by gromit.dlib.vt.edu (Postfix) with ESMTPSA id DAFD7518E3; Tue, 31 Jan 2023 16:12:58 -0500 (EST) Content-Type: text/plain; charset=us-ascii List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Slow WAN traffic to FreeBSD hosts but not to Linux hosts---how to debug/fix? From: Paul Mather In-Reply-To: <150b0a33-5695-6d12-d86c-0fba0b40d265@grosbein.net> Date: Tue, 31 Jan 2023 16:12:48 -0500 Cc: FreeBSD-STABLE Mailing List Content-Transfer-Encoding: quoted-printable Message-Id: <03DC7A96-7945-4C8B-A062-F8F00BD690C4@gromit.dlib.vt.edu> References: <95EDCFCA-7E3F-458F-85A6-856D606B9D98@gromit.dlib.vt.edu> <150b0a33-5695-6d12-d86c-0fba0b40d265@grosbein.net> To: Eugene Grosbein X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Rspamd-Queue-Id: 4P5ySX1G4Nz3Ptj X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:1312, ipnet:2001:468:c80::/48, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Jan 30, 2023, at 9:13 PM, Eugene Grosbein wrote: > 31.01.2023 4:17, Paul Mather wrote: >=20 >> TL;DR: When working from home, I can max out my residential 200 Mbit = network connection when downloading from remote Linux hosts at $JOB but = only manage about 20% of my max residential connection speed when = downloading from remote FreeBSD hosts at $JOB. When at $JOB, both = FreeBSD and Linux hosts have no problem saturating their GbE connections = transferring between each other. Why is this and how can I debug and = fix it? >>=20 >> I have a 200 Mbit residential cable connection (Xfinity, 200 Mbit = down/~10 Mbit up). I've noticed recently that I can easily get 10--20 = MB/s download speeds when transferring data from Linux hosts at work but = when I try to download that same data from the FreeBSD hosts I use the = speed usually tops out at 3--4 MB/s. These are Linux and FreeBSD hosts = that are on the same subnet at work. Transfers from the FreeBSD hosts = at work (within-subnet and within-site) are fine and match those of the = Linux hosts---often 112 MB/s. So, it just appears to be the traffic = over the WAN to my home that is affected. The WAN path from home to = this subnet is typically 15 hops with a typical average ping latency of = about 23 ms. >>=20 >> The FreeBSD hosts are a mixture of -CURRENT, 13-STABLE, and = 13.1-RELEASE. I had done some TCP tuning based upon the calomel.org = tuning document = (https://calomel.org/freebsd_network_tuning.html), but removed those = tuning settings when I noticed the problem but the problem still = persists. The only remaining customisation is that the 13-STABLE has = "net.inet.tcp.cc.algorithm=3Dcubic". (I notice that -CURRENT now has = this as default so wanted to try that on 13-STABLE, too.) The FreeBSD = systems are using either igb or em NICs. The Linux systems are using = similar hardware. None has a problem maintaining local GbE transfer = speeds---it's only the slower/longer WAN connections that have problems = for the FreeBSD hosts. >>=20 >> It seems that Linux hosts cope with the WAN path to my home better = than the FreeBSD systems. Has anyone else noticed this? Does anyone = have any idea as to what is obviously going wrong here and how I might = debug/fix the FreeBSD hosts to yield faster speeds? My workaround at = the moment is to favour using the remote Linux hosts for bulk data = transfers. (I don't like this workaround.) >>=20 >> Any help/insight is gratefully appreciated. >=20 > I bet speedy traffic does not cross any NAT boxes but perhaps you = employ NAT at your own place. > Both pfnat and ipfw nat are not compatible with TSO, also sometimes = RX/TX checksum offload for NIC produce broken checksums, > and all that creates excessive retransmissions and timeouts greatly = reducing traffic speed. My IPv4 clients at home are behind a NAT router (OPNsense 23.1 with HTCP = CC enabled and FQ_CoDel traffic shaper configured). The remote systems = at $JOB are not behind NAT. On my OPNsense router I have disabled = hardware offloading as (IIRC) these are generally not recommended for = routers. As such, I have "Disable hardware checksum offload"; "Disable = hardware TCP segmentation offload"; and "Disable hardware large receive = offload" all checked in Interfaces -> Settings in OPNsense. The home = router running OPNsense uses NICs that identify as "82583V Gigabit = Network Connection" (emX@pci0:X:0:0: class=3D0x020000 rev=3D0x00 = hdr=3D0x00 vendor=3D0x8086 device=3D0x150c subvendor=3D0x8086 = subdevice=3D0x0000). The FreeBSD systems at $JOB default to having TSO and RXCSUM/TXCSUM = enabled. I disabled these but it didn't make any apparent difference in = improving speeds. I looked at a Linux system on similar hardware and = "ethtool -k" indicates it also has TSO and RX/TX checksum offloading = enabled. > You may want to inspect traffic with Wireshark, as it shows = retransmissions and generally anomalies with colors, > or just go ahead and disable TSO and rxcsum/txcsum for external = interface. I'm going to pursue this next. I made Wireshark captures at the client = end but need to collect packet traces at the server side. Also, with = the help of your suggestion, I managed to find a colour configuration to = highlight retransmissions. Hopefully that might help differentiate the = FreeBSD vs. Linux situation. I'm not great at grokking Wireshark traces = but I guess I'm about to get better in the future. :-) Thanks for the suggestions. Cheers, Paul.