From nobody Sun Sep 29 12:46:57 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XGkVY5lPZz5XdMv; Sun, 29 Sep 2024 12:47:01 +0000 (UTC) (envelope-from void@f-m.fm) Received: from fout-a5-smtp.messagingengine.com (fout-a5-smtp.messagingengine.com [103.168.172.148]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4XGkVX4kHMz4TN1; Sun, 29 Sep 2024 12:47:00 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm2 header.b=bwVkHwgZ; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=HWwCQ0V9; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 103.168.172.148 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from phl-compute-07.internal (phl-compute-07.phl.internal [10.202.2.47]) by mailfout.phl.internal (Postfix) with ESMTP id 8BD4913804BD; Sun, 29 Sep 2024 08:46:59 -0400 (EDT) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-07.internal (MEProxy); Sun, 29 Sep 2024 08:46:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1727614019; x=1727700419; bh=5kyOTPA1Vk D2SjNsDWaZp2ksA/4KA+k3aKVE94il/NM=; b=bwVkHwgZz43nS5QtaLtvuDSKXF UZ7lQkkl6RrXkxQAIOCdWfQm26bPV5yb7TZpTj8vxlRZozmPMPM0fbMu1RYizzFe vuVG5DpubMXG+EyVNMeGHlZsS4Bgz0xTLNWeY4LWKrR6/hverpu9IBDBo5aQqO5K vyyqiT/EmeEZhVIQplzNG92XS9GDjHjyAOdzOAWzKSFGVgplTz7tNN0XOk3tZsZR 0U7rjS/7RE39ll9O62lFF4ETvob9pKeBUeXMh1VPpRUqrDlo0ZkI8T84ta8NnL0z bm3YjKlFbWd/0w/kNJmV8yxFYSWDWKMsOxF2d/DZUpVkJ/W4GDEmFn2+FOHg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1727614019; x=1727700419; bh=5kyOTPA1VkD2SjNsDWaZp2ksA/4K A+k3aKVE94il/NM=; b=HWwCQ0V9kDJGDGMcq43d2zrKM8qrdbq17PrYgGKJE3yn /7QIQlk9c7gk6eVp4e2eGF+oBarqF6mhkQsGF2troYQaNSAw009KIO0FFNFcupYD 6kpoZgDiE5QHUpuesWjYNJRf9qRFjs4mE2E7Ri9pibzZ7PAu3mOdXmzgF5d4v+dy XqPRtMJWp5Ls7F0naFycwX/ABn2s6yaENo8SBdFTcF/Ydf/r+yDjgzGoEEburCXW T458Z5s46SV3LWhyPVuFS2sL8Meq/iPsUXJA7ooE1TzxpSmENQHP9mMMuLy3DG/k yzs9a9KH8rSPiDwDjnduGiOUjSyDMx58Ct0LDdFguw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddrvddufedgheekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvve fukfhfgggtuggjsehttdertddttddvnecuhfhrohhmpehvohhiugcuoehvohhiugesfhdq mhdrfhhmqeenucggtffrrghtthgvrhhnpeduvdeugeduudefteehheffvdegteelkeekvd evffdukeeugeeuteffteefkeelhfenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgr mhepmhgrihhlfhhrohhmpehvohhiugesfhdqmhdrfhhmpdhnsggprhgtphhtthhopedvpd hmohguvgepshhmthhpohhuthdprhgtphhtthhopehfrhgvvggsshguqdhnvghtsehfrhgv vggsshgurdhorhhgpdhrtghpthhtohepfhhrvggvsghsugdqvhhirhhtuhgrlhhiiigrth hiohhnsehfrhgvvggsshgurdhorhhg X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 29 Sep 2024 08:46:58 -0400 (EDT) Date: Sun, 29 Sep 2024 13:46:57 +0100 From: void To: freebsd-net@freebsd.org Cc: freebsd-virtualization@freebsd.org Subject: Re: slow network performance in bhyve with freebsd guests compared with any other guest os Message-ID: Mail-Followup-To: freebsd-net@freebsd.org, freebsd-virtualization@freebsd.org References: List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: X-Spamd-Result: default: False [-3.79 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.986]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; MID_RHS_NOT_FQDN(0.50)[]; RWL_MAILSPIKE_VERYGOOD(-0.20)[103.168.172.148:from]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm2,messagingengine.com:s=fm2]; R_SPF_ALLOW(-0.20)[+ip4:103.168.172.128/27]; RCVD_IN_DNSWL_LOW(-0.10)[103.168.172.148:from]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_FROM(0.00)[f-m.fm]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_THREE(0.00)[3]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; TO_DN_NONE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org,freebsd-virtualization@freebsd.org]; ASN(0.00)[asn:209242, ipnet:103.168.172.0/24, country:US]; RCVD_VIA_SMTP_AUTH(0.00)[]; MISSING_XM_UA(0.00)[]; DWL_DNSWL_NONE(0.00)[messagingengine.com:dkim] X-Rspamd-Queue-Id: 4XGkVX4kHMz4TN1 X-Spamd-Bar: --- On Sun, Sep 29, 2024 at 08:12:23AM +0100, Lexi Winter wrote: >On 29/09/2024 07:58, void wrote: >>Surprisingly, freebsd guest performance is about 1/3rd of the line speed. >>Do some sysctls need to be tuned in freebsd specifically for when it >>is in a guest context? > >i tested this here and cannot reproduce the problem: The iperf3 performance is appears to be heavily influenced by load. The linux guest still massively outperforms the freebsd guests The tests I posted before were when the bhyve host wasn't under much load. 0.00 to 1.00. Right now, load is like uptime 1:38p.m. up 4:14, 2 users, load averages: 33.66, 33.47, 33.55 and iperf3 performance for freebsd 14-p5 guest is [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-30.25 sec 143 MBytes 39.8 Mbits/sec 5 sender [ 5] 0.00-30.25 sec 143 MBytes 39.8 Mbits/sec receiver and performance for ubuntu guest is [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-30.04 sec 778 MBytes 217 Mbits/sec 78 sender [ 5] 0.00-30.11 sec 776 MBytes 216 Mbits/sec receiver I think it might be due to mabe interrupts, epoll (which i think is a linux thing), maybe queueing. But I dont know how to fix it. Testing from the bhyve host (so no bhyve involvement) is line speed regardless of load. Maybe it's the way bhyve handles interrupts? --