From nobody Mon Aug 14 21:35:07 2023 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RPnkH0kYHz4mPfl for ; Mon, 14 Aug 2023 21:35:19 +0000 (UTC) (envelope-from SRS0=6rU7=D7=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RPnkG5B9xz4WXs for ; Mon, 14 Aug 2023 21:35:18 +0000 (UTC) (envelope-from SRS0=6rU7=D7=quip.cz=000.fbsd@elsa.codelab.cz) Authentication-Results: mx1.freebsd.org; none Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id D087CD7899; Mon, 14 Aug 2023 23:35:08 +0200 (CEST) Received: from [192.168.145.49] (ip-89-177-27-225.bb.vodafone.cz [89.177.27.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id DC491D788E; Mon, 14 Aug 2023 23:35:07 +0200 (CEST) Message-ID: <6bb84230-7bee-497e-4076-88a2603acb40@quip.cz> Date: Mon, 14 Aug 2023 23:35:07 +0200 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Subject: Re: bhyve VM not getting as much upload bandwidth as the host To: "Patrick M. Hausen" Cc: Odhiambo Washington , freebsd-virtualization@freebsd.org References: <096b2d73-b28c-bd6b-a6d3-2a3c3dbaea7e@h4ck.io> <38587cf9-29a7-1246-3b64-a3cf35933717@quip.cz> <96160FBF-4E48-48E3-B26C-026CEB5B03C2@punkt.de> Content-Language: cs-Cestina, en-US From: Miroslav Lachman <000.fbsd@quip.cz> In-Reply-To: <96160FBF-4E48-48E3-B26C-026CEB5B03C2@punkt.de> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4RPnkG5B9xz4WXs X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:42000, ipnet:94.124.104.0/21, country:CZ] On 14/08/2023 11:38, Patrick M. Hausen wrote: [..] >> I never had this solved. Even my FreeBSD guest on FreeBSD host with VirtualBox is slow as few hunderds kB/s >> It was like 10Mbps with Bhyve. I only use VMs for testing but installing packages is always so slow. So you are not alone. I would really like to know how to improve the network speed with virtualized environment. > > This looks weird to me. I run lots of VMs in production on TrueNAS CORE > - essentially FreeBSD 13-STABLE with bhyve and all of them get near gigabit > speed with bridged networking. > > Guests: > > Windows > Ubuntu > FreeBSD (OPNsense) > > Specifically the OPNsense VM can route 700-800 Mbit/s across gigabit interfaces. > > All my VMs use VirtIO network drivers - do yours? NICs are emulated Intel PRO/1000 MT but I tested VirtIO in the past and it was the same. Now I tried to disable RXCSUM and TXCSUM on host interface and it fixed it. It is strange to me because I never had speed problem on the host it self so I do not suspect anything wrong on host's interface. 1) speedtest in VM with RXCSUM & TXCSUM enabled Hosted by Rohwerder Datasystems GmbH (Schwarzenbek) [463.71 km]: 42.018 ms Testing download speed Download: 7.37 Mbit/s Testing upload speed Upload: 25.16 Mbit/s 2) speedtest in VM right after I disabled RXCSUM & TXCSUM on the host Hosted by Rohwerder Datasystems GmbH (Schwarzenbek) [463.71 km]: 23.841 ms Testing download speed Download: 242.99 Mbit/s Testing upload speed Upload: 24.80 Mbit/s 250 Mbit/s is a full speed of my home internet connection. I don't know why but RXCSUM and TXCSUM cause more problems in my experiences. I remember I must disabled RXCSUM & TXCSUM in completely virtual environment of some cloud provider (based on KVM & Qemu) where we run about 10 machines and one of them should act like an Ipsec gateway fore a couple of private VMs. Packets were not forwarded between 2 NICs until i disabled RX & TX csums on private VirtIO NIC - all VMs were FreeBSD 12.3 at that time. Thank you! Best regards Miroslav Lachman