From nobody Tue Oct 15 02:48:56 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XSJTM0rH4z5NkJR for ; Tue, 15 Oct 2024 02:49:11 +0000 (UTC) (envelope-from void@f-m.fm) Received: from fout-a4-smtp.messagingengine.com (fout-a4-smtp.messagingengine.com [103.168.172.147]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4XSJTK66Xtz4SSR for ; Tue, 15 Oct 2024 02:49:09 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm2 header.b=vc38O8Tf; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=WJxjT2NP; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 103.168.172.147 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from phl-compute-10.internal (phl-compute-10.phl.internal [10.202.2.50]) by mailfout.phl.internal (Postfix) with ESMTP id 412F91380604 for ; Mon, 14 Oct 2024 22:49:09 -0400 (EDT) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-10.internal (MEProxy); Mon, 14 Oct 2024 22:49:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1728960549; x=1729046949; bh=z90uHGZrn+ oRQf+lqyZrkOs8ZvuM0rPj17ZUolSOcJM=; b=vc38O8TfNCLDj0bDHCDNQAjO6i m6oD4t5HiIP6oIdIhAdQEUsOFvfHdrGqU9FL0GpNBcTSi/kXW10PVmknHWHza5PF 9yWVUu/mQ17qeWgaH1OYzDFpOtNXE989fiE8+xjVTm7ws42A2WHWURrfIRM9Zu32 el3RYwT0UNZYcqSlP2SD+kfoeEdxwUgLFrXMyThPmtba2pnv4k+tj3V4KUVFOncv cyp8sX00Y/WnOm1Qjf6y1Rt5sfnUaezRwT8jOQo0mTKTtKpQt6gUJ7nF2FEKw1nI OA9ORIFxQ4Yp1ZLC+mNHuf3/pHZ56AXoiUQJ9me4vzlZmbRsuDQ9Lr+BH2jQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1728960549; x=1729046949; bh=z90uHGZrn+oRQf+lqyZrkOs8ZvuM 0rPj17ZUolSOcJM=; b=WJxjT2NPOtFWBbh0ATeJOXQK7sJiAdSEJZrZCW6zOx5c XdB70Hvpt32DBxbOU7dhmV/yUNWSONiCn2X19HKN4nmd00hciFLGoltUuYDFG1yV m/0aZVC1/MZIUxiDmaIVYb0GUy86tgtk4NyaQ9bau3PxmBWrsPaOM/0m8KvHQ2TI Eo7Zho9FeienMCemG5RybfIquTNywW8CyQZ0HdENnqennFT6N2xj4dB/8s9QTod1 NzIP6sPl7KoxEN6YG5vZALrKlAvpH6fUTon5IGdVVSgtemeD1M4sglOs2OjFxjQ4 SeZfsfPRbhA3okuFDemvG4ZobHNufAkSQrW5KMhDWQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddrvdegiedgieegucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuf fkfhggtggujgesthdtredttddtvdenucfhrhhomhepvhhoihguuceovhhoihgusehfqdhm rdhfmheqnecuggftrfgrthhtvghrnhepleevvefhheeffeffheefffekvedvfffgtdffje fhfedvgeelveevudejjeejleehnecuffhomhgrihhnpehushgvrhdrfhhmnecuvehluhhs thgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepvhhoihgusehfqdhmrd hfmhdpnhgspghrtghpthhtohepuddpmhhouggvpehsmhhtphhouhhtpdhrtghpthhtohep fhhrvggvsghsugdqnhgvthesfhhrvggvsghsugdrohhrgh X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Mon, 14 Oct 2024 22:49:08 -0400 (EDT) Date: Tue, 15 Oct 2024 03:48:56 +0100 From: void To: freebsd-net@freebsd.org Subject: Re: Performance issues with vnet jails + epair + bridge Message-ID: References: <20240912181618.7895d10ad5ff2ebae9883192@gmail.com> List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: <20240912181618.7895d10ad5ff2ebae9883192@gmail.com> X-Spamd-Result: default: False [-3.59 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.995]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; MID_RHS_NOT_FQDN(0.50)[]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm2,messagingengine.com:s=fm2]; R_SPF_ALLOW(-0.20)[+ip4:103.168.172.128/27]; RCVD_IN_DNSWL_LOW(-0.10)[103.168.172.147:from]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[f-m.fm]; MIME_TRACE(0.00)[0:+]; FROM_HAS_DN(0.00)[]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; TO_DN_NONE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[103.168.172.147:from]; MISSING_XM_UA(0.00)[]; ASN(0.00)[asn:209242, ipnet:103.168.172.0/24, country:US]; DWL_DNSWL_NONE(0.00)[messagingengine.com:dkim] X-Rspamd-Queue-Id: 4XSJTK66Xtz4SSR X-Spamd-Bar: --- On Thu, Sep 12, 2024 at 06:16:18PM +0100, Sad Clouds wrote: >Hi, I'm using FreeBSD-14.1 and on this particular system I only have a >single physical network interface, so I followed instructions for >networking vnet jails via epair and bridge, e.g. (snip) >The issue is bulk TCP performance throughput between this jail and the >host is quite poor, with one CPU spinning 100% in kernel and others >sitting mostly idle. > >It seems there is some lock contention somewhere, but I'm not sure if >this is around vnet, epair or bridge subsystems. Are there >other alternatives for vnet jails? Can anyone recommend specific >deployment scenarios? I've seen references to netgraph which could be >used with jails. Does it have better performance and scalability and >could replace epair and bridge combination? I've noticed bandwidth problems in virtualised adapters, too. I ran some simple tests and put the results here: http://void.f-m.fm.user.fm/bhyve-virtio-testing.html My own context here is bhyve vms. Linux guests greatly out-perform FreeBSD ones and I'm trying to find out why, if it's a tunable that needs tuning, if it's a fault with bge0, how it could be fixed. It's interesting to me that you see similar effects in quite a different context. I'm using bridge and tap interfaces and within the (freebsd) vms the interface is vtnet0. So maybe there's something amiss or needs tuning on these virtual interfaces? The bhyve host gets line speeds after accounting for tcp/ip overhead, as expected. It's just the vms. I've read that linux doesn't "epoll" something like that but I don't know much of anything about linux. Clearly it's doing something different with its own virtualised adapter, internally. --