From nobody Sun Sep 29 06:58:43 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XGZml4cJQz5XF4k; Sun, 29 Sep 2024 06:58:47 +0000 (UTC) (envelope-from void@f-m.fm) Received: from fhigh-a8-smtp.messagingengine.com (fhigh-a8-smtp.messagingengine.com [103.168.172.159]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4XGZmk5tp2z4rlr; Sun, 29 Sep 2024 06:58:46 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm2 header.b=cPmU8+Td; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=l+RfyJT8; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 103.168.172.159 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from phl-compute-06.internal (phl-compute-06.phl.internal [10.202.2.46]) by mailfhigh.phl.internal (Postfix) with ESMTP id 677AF11402B4; Sun, 29 Sep 2024 02:58:45 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-06.internal (MEProxy); Sun, 29 Sep 2024 02:58:45 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :message-id:mime-version:reply-to:subject:subject:to:to; s=fm2; t=1727593125; x=1727679525; bh=HTcPtZe+V3Y0O2vz6NqBiVdfzHJaVkJO y9UzsUbz5LA=; b=cPmU8+Td9J02OTIwUNGEUrJm1Ti/CuS4p+R1786+2UKQtrLF MQPmFMUJZL0Ax9iDwhaUBIV8W6vIJpO+JyjP8RttJWOZJ3hZvRp5z7Ydybf76pUw hxGLeRyzG1/Ag1AZ2CR475u/LlIDsIzQ5G7es6Y3/lQGVsf8HO/9ouBghrVR/P+D l4gt9T55I3IVKOVlykqCjeqP0vEmb1/4IXG1YNV1H2bvmZFQZ9EyQiXj5vfXuNby UsQ7adz+M82fuHB4MjnlLGA7Fh8c8WbvlDw92CGLoN8So6UZvmrQU2IB1IXNtzkQ 5PvVHDiG77LkKnl8Qgv6Xu/oWvgHB76xowB21A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:message-id :mime-version:reply-to:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t= 1727593125; x=1727679525; bh=HTcPtZe+V3Y0O2vz6NqBiVdfzHJaVkJOy9U zsUbz5LA=; b=l+RfyJT8CQv1lG/GdUWEJSECIMVLT0teG7/c21kB/CbYSU6pwyj Wm7Opow2jnPBftSNU7f7DWD5eRBgpYrnOV9MxeYML3OJivg05DhcdjxhUMXn7moh 32tXaJJ/pQGP/vqRJsFMboiaQqpuG/qzaR23HcAy9OgNIYX6nGYh7qbAT20QFKeV wrz7skaWdu+IX3skr0nrkYq0oqjunT2n8LdB9qAT2vqD4I3gLgybbqkClA/lUffC +c3dFo/MA7t5xP74XeNoF51WstcsvOkMvzCuCR9xIfU2r5Wq5sYFLVlV8BOUFGUd xi71q0N30wIF1wdlaAVVM7wUmGfUW2dsUAg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddrvdduvddgudduiecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecunecujfgurhepfffhvf evuffkgggtugesthdtredttddtvdenucfhrhhomhepvhhoihguuceovhhoihgusehfqdhm rdhfmheqnecuggftrfgrthhtvghrnhepueekjefhleduiedttefhffejieeugfehtdefle evgfduieeuvedviedujefhtdelnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghm pehmrghilhhfrhhomhepvhhoihgusehfqdhmrdhfmhdpnhgspghrtghpthhtohepvddpmh houggvpehsmhhtphhouhhtpdhrtghpthhtohepfhhrvggvsghsugdqnhgvthesfhhrvggv sghsugdrohhrghdprhgtphhtthhopehfrhgvvggsshguqdhvihhrthhurghlihiirghtih honhesfhhrvggvsghsugdrohhrgh X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 29 Sep 2024 02:58:44 -0400 (EDT) Date: Sun, 29 Sep 2024 07:58:43 +0100 From: void To: freebsd-net@freebsd.org Cc: freebsd-virtualization@freebsd.org Subject: slow network performance in bhyve with freebsd guests compared with any other guest os Message-ID: Mail-Followup-To: freebsd-net@freebsd.org, freebsd-virtualization@freebsd.org List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline X-Spamd-Result: default: False [-3.54 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.94)[-0.942]; MID_RHS_NOT_FQDN(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm2,messagingengine.com:s=fm2]; R_SPF_ALLOW(-0.20)[+ip4:103.168.172.128/27]; RCVD_IN_DNSWL_LOW(-0.10)[103.168.172.159:from]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_FROM(0.00)[f-m.fm]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_THREE(0.00)[3]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; TO_DN_NONE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org,freebsd-virtualization@freebsd.org]; MISSING_XM_UA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:209242, ipnet:103.168.172.0/24, country:US]; DWL_DNSWL_NONE(0.00)[messagingengine.com:dkim] X-Rspamd-Queue-Id: 4XGZmk5tp2z4rlr X-Spamd-Bar: --- Surprisingly, freebsd guest performance is about 1/3rd of the line speed. Do some sysctls need to be tuned in freebsd specifically for when it is in a guest context? The host is 15.0-CURRENT (GENERIC-NODEBUG) #1 n271832-04262ed78d23 Xeon E5-2690 @ 2.90GHz with 128GB RAM and the guests are all on a ssd zpool. iperf3 from the guests to a freebsd 14-stable workstation via a 1G switch. The network is all 1GB iperf3 --time 30 -c desktop freebsd15-testvm guest ====================== [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-30.05 sec 1.36 GBytes 389 Mbits/sec 0 sender [ 5] 0.00-30.05 sec 1.36 GBytes 389 Mbits/sec receiver then, with kldload tcp_rack and then sysctl net.inet.tcp.functions_default=rack [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-30.01 sec 1.47 GBytes 420 Mbits/sec 0 sender [ 5] 0.00-30.02 sec 1.47 GBytes 420 Mbits/sec receiver freebsd-14.1-p5 vm guest ======================== [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-30.07 sec 1.19 GBytes 340 Mbits/sec 0 sender [ 5] 0.00-30.07 sec 1.19 GBytes 340 Mbits/sec receiver then, with kldload tcp_rack and then sysctl net.inet.tcp.functions_default=rack [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-30.03 sec 1.09 GBytes 313 Mbits/sec 0 sender [ 5] 0.00-30.04 sec 1.09 GBytes 313 Mbits/sec receiver tcp_rack seems to have made it slower in 14-p5, quicker in -current from bhyve host, baremetal ========================== [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-30.01 sec 3.29 GBytes 941 Mbits/sec 3 sender [ 5] 0.00-30.01 sec 3.29 GBytes 941 Mbits/sec receiver compare with (from) Ubuntu 24.04.1 LTS guest: ============================================= [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-30.00 sec 3.27 GBytes 935 Mbits/sec 0 sender [ 5] 0.00-30.01 sec 3.27 GBytes 935 Mbits/sec receiver alpine linux guest: =================== [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-30.01 sec 3.29 GBytes 941 Mbits/sec 0 sender [ 5] 0.00-30.01 sec 3.29 GBytes 941 Mbits/sec receiver --