From nobody Thu Jun 29 11:46:00 2023 X-Original-To: freebsd-transport@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QsGqc5pqdz4kXHc for ; Thu, 29 Jun 2023 11:46:04 +0000 (UTC) (envelope-from rrs@netflix.com) Received: from mail-pg1-x52d.google.com (mail-pg1-x52d.google.com [IPv6:2607:f8b0:4864:20::52d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QsGqc4XN7z3qGk for ; Thu, 29 Jun 2023 11:46:04 +0000 (UTC) (envelope-from rrs@netflix.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pg1-x52d.google.com with SMTP id 41be03b00d2f7-55b1238cab4so312278a12.2 for ; Thu, 29 Jun 2023 04:46:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netflix.com; s=google; t=1688039163; x=1690631163; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=wghCVvSr25nh1VMnd282hqjxtVfrg9k40NQkh/Fce+I=; b=pdXo36hx59Q+v9B0Tm2rnwSljdVTfnZc92xAk/D3H+uCneOUiCV7kkITixBnxmxpNk i9Lv7kOeze7HewiWwKGSvI7TsAFSAHhuokx+ZfLU/PDjwM4zmc50fihjs40mOLzKZ9ga rZ5EuTA/ZtDXvMfzK2PjIyFvXzhCZmAtJgpTQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688039163; x=1690631163; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wghCVvSr25nh1VMnd282hqjxtVfrg9k40NQkh/Fce+I=; b=GB3aSnQPAmSzLMzaaWpJvhK1tzsDBkNmkbVSwvubT1WqXWQlrcA3zFnF6KYOsio7WT c7RbhgJzJUnAXGACDJcDoAIACdhy1XDSpLDgkzOnEjUdMdStFLWPtRzJfDFTcIvj/Bgi R+pXWHXSkdGqyURuSTDBl8kXZvt9bZFcqwkXxhYm3JKds7A+BzSGMp6Tlm5xnepkgXaU Ei1PnpolxCVHoqCiNjgwDRcgRmcDYZFiL082iYqJtQH+vOjiNQf5ngfp4aVcLVCFJmRg jN+VO6muamioWo3bYxUQLebtBpdciphdxa0BU0IiH+fqLwRSBZ/thWNWHKP0YifWYoeM Wrgw== X-Gm-Message-State: AC+VfDx/d+qJk/iswH/LbWeUcnmxb3gbWkEkDjcJZwasMeqws4ZTGUKR xgVu4NHIxahFyqtbdnICSflz2/SNr2dkzVii8p4= X-Google-Smtp-Source: ACHHUZ4uIoaqyP7JbyAs31aVpwq/XDDrNzobDpaIqzY98mQNzJ4lMW3altS22KubJ8ZlSfufQCR2iw== X-Received: by 2002:a05:6a20:7f8a:b0:118:2e5a:efab with SMTP id d10-20020a056a207f8a00b001182e5aefabmr37205977pzj.4.1688039163152; Thu, 29 Jun 2023 04:46:03 -0700 (PDT) Received: from smtpclient.apple (072-239-139-254.res.spectrum.com. [72.239.139.254]) by smtp.gmail.com with ESMTPSA id x18-20020a170902ec9200b001b66a71a4a0sm4171361plg.32.2023.06.29.04.46.01 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 29 Jun 2023 04:46:02 -0700 (PDT) From: Randall Stewart Message-Id: <089C6BBB-C978-46DE-9479-1285B112E90D@netflix.com> Content-Type: multipart/signed; boundary="Apple-Mail=_F3B8C638-6298-4CA3-B37E-159AD1A4F036"; protocol="application/pkcs7-signature"; micalg=sha-256 List-Id: Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-transport List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-transport@freebsd.org X-BeenThere: freebsd-transport@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.3\)) Subject: Re: FreeBSD TCP (with iperf3) comparison with Linux Date: Thu, 29 Jun 2023 07:46:00 -0400 In-Reply-To: Cc: "freebsd-transport@FreeBSD.org" To: Murali Krishnamurthy References: X-Mailer: Apple Mail (2.3696.120.41.1.3) X-Rspamd-Queue-Id: 4QsGqc4XN7z3qGk X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_F3B8C638-6298-4CA3-B37E-159AD1A4F036 Content-Type: multipart/alternative; boundary="Apple-Mail=_D4C7D065-0C42-45C8-9FF3-05D3B02AB707" --Apple-Mail=_D4C7D065-0C42-45C8-9FF3-05D3B02AB707 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Greetings Murali: So I am unclear from your =E2=80=9CSocket buffer=E2=80=9D as to what you = did.. Did you set both the send and receive windows to 16Mbps with the SO_SNDBUF and = SO_RCVBUF options? Or were you just using auto-scaling to have the socket buffers advance? Thanks R > On Jun 29, 2023, at 5:51 AM, Murali Krishnamurthy = wrote: >=20 > Hello FreeBSD Transport experts, > =20 > We are evaluating performance of FreeBSD 13 VM on ESX hypervisor in = long RTT setup and happened to compare the performance with Linux VM = with same hypervisor. > We see a substantially better performance with Linux getting close to = the BDP limit, whereas BSD 13 not filling up the pipe enough. > We are trying to figure out what could lead to such a huge difference = and feel we could be missing something here. > =20 > Could you please help us to know if there is a way to make it perform = better? > =20 > Setup details: > =20 > We have 2 ESX hypervisors where 2 VMs (one FreeBSD 13 and one Ubuntu = 23.04/Linux kernel 6.2) were launched on each hypervisor. > Then we ran iperf between, > BSD 13 <-> BSD 13 > Ubuntu <-> Ubuntu > =20 > Even though the network environment were same in both cases, we see = Ubuntu performing much better. > =20 > Below are connection parameters: > Socket buffer: 16MB > TCP CC Algo: Cubic. We used this as this is suitable for Long Fat = Networks. > Ping RTT: 100 ms between the two end points. > We kept all other parameters to default on both Linux and BSD. > =20 > BDP for 16MB Socket buffer: 16 MB * (1000 ms * 100ms latency) * 8 = bits/ 1024 =3D 1.25 Gbps > =20 > Ubuntu consistently hits around 1 Gbps Bitrate almost reaching the BDP = limit. > FreeBSD 13 shows a Bit rate between the range of 300-600 Mbps only. So = it seems to be doing half as good as Linux. > For lower socket buffer of 4MB, both FreeBSD and Linux seem to do same = and able to meet BDP of 300 Mbps consistently. > Larger socket buffer seems to have an issue. > =20 > Please let us know if there are ways to fine tune the system = parameters to make BSD perform better. > Or any other suggestions/queries welcome. > =20 > Regards > Murali ------ Randall Stewart rrs@netflix.com --Apple-Mail=_D4C7D065-0C42-45C8-9FF3-05D3B02AB707 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Greetings Murali:

So I am unclear from your =E2=80=9CSocket buffer=E2=80=9D as = to what you did.. Did you
set both the send and = receive windows to 16Mbps with the SO_SNDBUF and SO_RCVBUF = options?
Or were you just using auto-scaling to = have the socket buffers advance?

Thanks

R

On Jun = 29, 2023, at 5:51 AM, Murali Krishnamurthy <muralik1@vmware.com>= wrote:

Hello FreeBSD = Transport experts,
 
We are = evaluating performance of FreeBSD 13 VM on ESX hypervisor in long RTT = setup and happened to compare the performance with Linux VM with same = hypervisor.
We see a substantially better performance with = Linux getting close to the BDP limit, whereas BSD 13 not filling up the = pipe enough.
We are trying to figure out what could lead to = such a huge difference and feel we could be missing something here.
 
Could you please help us to know if there is a = way to make it perform better?
 
Setup details:
 
We have 2 ESX hypervisors where 2 VMs (one = FreeBSD 13 and one Ubuntu 23.04/Linux kernel 6.2) were launched on each = hypervisor.
Then we ran iperf between,
  1. BSD = 13 <-> BSD 13
  2. Ubuntu = <-> Ubuntu
 
Even though the network environment were same = in both cases, we see Ubuntu performing much better.
 
Below are connection parameters:
Socket buffer: 16MB
TCP CC Algo: = Cubic. We used this as this is suitable for Long Fat Networks.
Ping RTT:  100 ms between the two end points.
We kept all other parameters to default on both Linux and = BSD.
 
BDP for = 16MB Socket = buffer: 16 MB * (1000 ms * 100ms = latency) * 8 bits/ 1024 =3D 1.25 = Gbps
 
Ubuntu consistently hits around 1 Gbps Bitrate = almost reaching the BDP limit.
FreeBSD 13 shows = a Bit rate between the range of 300-600 = Mbps only. So it seems to be doing half = as good as Linux.
For lower socket buffer of 4MB, both FreeBSD = and Linux seem to do same and able to meet BDP of 300 Mbps = consistently.
Larger socket buffer seems to have an = issue.
 
Please let us = know if there are ways to fine tune the system parameters to make BSD = perform better.
Or any other suggestions/queries welcome.
 
Regards
Murali

------
Randall = Stewart



= --Apple-Mail=_D4C7D065-0C42-45C8-9FF3-05D3B02AB707-- --Apple-Mail=_F3B8C638-6298-4CA3-B37E-159AD1A4F036 Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCAzYw ggMyMIICGqADAgECAgqxywKqrHPB2ybTMA0GCSqGSIb3DQEBCwUAMEcxGDAWBgNVBAMTD1JhbmRh bGwgU3Rld2FydDEeMBwGCSqGSIb3DQEJARYPcnJzQG5ldGZsaXguY29tMQswCQYDVQQGEwJVUzAe Fw0yMTAxMjQxMjIwMTRaFw0yNjAxMjQxMjIwMTRaMEcxGDAWBgNVBAMTD1JhbmRhbGwgU3Rld2Fy dDEeMBwGCSqGSIb3DQEJARYPcnJzQG5ldGZsaXguY29tMQswCQYDVQQGEwJVUzCCASIwDQYJKoZI hvcNAQEBBQADggEPADCCAQoCggEBAMUAht2nr/NFlK+tmmN9PdO3DBPfeYh9fLcbVihR+/dipO41 AsFy9y+2uDVaFhTEvp406P0o9PQQTuYXqrCr76eWQIj3V787e1WKjTup1mIyQeWHGf1gvb/7vmI2 zHg6QZEIC4W8xeO8SLKyHiwlFHZn8Rn1HxtB7Ge+NulygkgUgJYhXD5E29jVGXAc6Qn9Vr9AexPf KaOhHCaNB/Twcinayz6D8CO/Ym1LOs3+ceSOa4cB07fepmbqDSXDkOeA3U7KLaluHrRTlj6DO+JU nqKXX7jJ68KTYSZ0qH4fZsk8cxFkwYI/3HDJi+oF+FDkf7SRo1Q2w+e3M/5MReLIQ7sCAwEAAaMg MB4wDwYJKoZIhvcvAQEKBAIFADALBgNVHQ8EBAMCB4AwDQYJKoZIhvcNAQELBQADggEBAHJfum1j 1WIVFjOJT/hqMIN751aXkablmwesW94lNJKjslPULbbcP5nZGg2lGpHcZ+0I5F/1TTiEsT2H2rhA uAnSsUxTpxRA+aoe+xtJOa5vle3CprhFkHAvB7EIoLiNaPd0DNK6kKYsbvr5Z5Eq7TF3SIO77Qh9 /8VgUfOb0ARDgix59Q6MM0NmIabEwh0cDWQYlGgDDtN9DNk5PGM4pjs48suwEdLmFTMOmGTkCp7I Vq6iHDNinBiB6+BB4VYMAO1o9qS+0pnfdmPJybt0zVGrhm/c1Fmm3Jec7NEuiKeXmhPIwdwMkKyp AsX0sHuFCYwioBTHHZpvnir+H2rRakgxggHrMIIB5wIBATBVMEcxGDAWBgNVBAMTD1JhbmRhbGwg U3Rld2FydDEeMBwGCSqGSIb3DQEJARYPcnJzQG5ldGZsaXguY29tMQswCQYDVQQGEwJVUwIKscsC qqxzwdsm0zANBglghkgBZQMEAgEFAKBpMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZI hvcNAQkFMQ8XDTIzMDYyOTExNDYwMFowLwYJKoZIhvcNAQkEMSIEILaabTtURq4atG0MExNLW/wO 8Ouv2lkNCnBERFs6wJwSMA0GCSqGSIb3DQEBCwUABIIBAIdtF+DXNOHZ2pJB1YkkLSaNc3uTedsy w45aDrz//YcaClhr/KWJnZ2vStcGv1O6tqXQEmfXPYACQFB/iOwsIahe8NjY71ysxl1f+37JEmnY 1GlOEIkjSoHr7JPmPDvRGtbaVQWsjeLDXJ5fZ1cUOmKB2h1g+sBdD4fTITRfN9z2JTDjWkVlSdw0 BvI1H+jke17uLoGLqVtPPNyW3NpWoKaTWz+1Ug7Bk/Qkz14Yhcq8elF0w+IZ/ySZQc32j9Hhjwl/ xOFH/xpyxgNx8yFEOJFNpwurBpGvfO14uEAzAxJ8DZva9sn6C5VfoQKa+ApkYlvo2xCib/tlT2on tra52DwAAAAAAAA= --Apple-Mail=_F3B8C638-6298-4CA3-B37E-159AD1A4F036--