From nobody Mon Aug 14 11:32:08 2023 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RPXM32z4tz4qMn3 for ; Mon, 14 Aug 2023 11:32:47 +0000 (UTC) (envelope-from odhiambo@gmail.com) Received: from mail-lj1-x231.google.com (mail-lj1-x231.google.com [IPv6:2a00:1450:4864:20::231]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RPXM24z4Cz4Xwf for ; Mon, 14 Aug 2023 11:32:46 +0000 (UTC) (envelope-from odhiambo@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=ahFPVyqy; spf=pass (mx1.freebsd.org: domain of odhiambo@gmail.com designates 2a00:1450:4864:20::231 as permitted sender) smtp.mailfrom=odhiambo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-lj1-x231.google.com with SMTP id 38308e7fff4ca-2b9c907bc68so61125861fa.2 for ; Mon, 14 Aug 2023 04:32:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692012765; x=1692617565; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=+tVpoQtiomlOQdUAnAxg+J84VJwo/JYmYv9dTG9IIlk=; b=ahFPVyqyLJPds0+sek4mPS0PI7iT4R5ZO5YbVY9Y79jrX5EBC4Z6Z+TWsJLLe4bJmM jdmCiCORPqAqbdtytrTZ3VkraDhcHOv37xEIF/KcbXxhSgSCf1/Ij26kodNDszLZn8wc jkLa3EKVV4U55BgGzQ7X0rPhfHJokoJ2R8qQIxPfe9Xq5haYVa48JEkjBz5Cr0j3lpeZ YFJYTQrnu5PDv96DFkD1HhBT49JQTvLpbZbcB6Z+dzp38o4eLgqBshh2TOaUmauafl6+ lkRs0n/kA5mPTUW1kmb3tmIFaAPa6VeRZIhPtaIl9QxgjA6I4I9nljJ0a1ewDKx0J6Sz ZWXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692012765; x=1692617565; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+tVpoQtiomlOQdUAnAxg+J84VJwo/JYmYv9dTG9IIlk=; b=Cr6QBtir4WKx9VMOeBQpr5e8ysFb4j9Z4qhFcQ/eiQ2kTtfnuk6/J4ppH3J4BnGs21 ZS09E8FGiyL9G2Dfei24wghF6rMyXX83HfammjbqXnu06lSKjFsjaSPq1+ntwi9VaaH7 UDw7hFzvZsRKMLIWYGahd/2LMAn+LSZJT+M7ZPLBV6GyJ4ADCF521KnRyXW1098hy86l z5plRiwo/noxU2RZ0+YjF2EyKGm8pndiiSi7qr70+garPoJGh/2u8tVPefqyTvEsD7Fh EBkXlIu9R+Etfn42JH7DfdRXIz+6ZZga5DvvwmegwQfcX1ARUwN0DLz66gIwFirmF4VH hbOg== X-Gm-Message-State: AOJu0YyK3D9Yn6qAmqLK1P47ZZy0+vOEzT6n8ZU8dELCKzWkNiI1RSCt aPBaY0wl8qca8V1f+JZRuYWsEn5hIGI5cHTm+zOjCX1FtBydgQ== X-Google-Smtp-Source: AGHT+IExS2xVOE7mBdRPhA4uw3iEd8X75SJPRvqagfhAI6AumPId/iCf1qTHfQpzQCvUecWR/c9agYoNqKVhPvGC82U= X-Received: by 2002:a2e:b057:0:b0:2b9:acad:b4b2 with SMTP id d23-20020a2eb057000000b002b9acadb4b2mr6471508ljl.6.1692012764441; Mon, 14 Aug 2023 04:32:44 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 References: <096b2d73-b28c-bd6b-a6d3-2a3c3dbaea7e@h4ck.io> <38587cf9-29a7-1246-3b64-a3cf35933717@quip.cz> <96160FBF-4E48-48E3-B26C-026CEB5B03C2@punkt.de> In-Reply-To: <96160FBF-4E48-48E3-B26C-026CEB5B03C2@punkt.de> From: Odhiambo Washington Date: Mon, 14 Aug 2023 14:32:08 +0300 Message-ID: Subject: Re: bhyve VM not getting as much upload bandwidth as the host To: freebsd-virtualization@freebsd.org Content-Type: multipart/alternative; boundary="000000000000c7b1450602e06b7e" X-Spamd-Result: default: False [-1.66 / 15.00]; HTTP_TO_IP(1.00)[]; URI_COUNT_ODD(1.00)[9]; NEURAL_HAM_LONG(-0.99)[-0.993]; NEURAL_HAM_SHORT(-0.96)[-0.960]; NEURAL_HAM_MEDIUM(-0.70)[-0.705]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::231:from]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_DN_NONE(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; RCVD_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org] X-Spamd-Bar: - X-Rspamd-Queue-Id: 4RPXM24z4Cz4Xwf --000000000000c7b1450602e06b7e Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, Aug 14, 2023 at 12:38=E2=80=AFPM Patrick M. Hausen wrote: > Hi all, > > > Am 14.08.2023 um 11:30 schrieb Miroslav Lachman <000.fbsd@quip.cz>: > > > > On 14/08/2023 10:49, Odhiambo Washington wrote: > >> I had the following in sysctl.conf: > >> net.link.tap.up_on_open=3D1 > >> net.link.bridge.pfil_member=3D0 > >> net.link.bridge.pfil_bridge=3D0 > >> net.link.bridge.pfil_local_phys=3D1 > >> So I only did: > >> sysctl -w net.link.bridge.pfil_onlyip=3D0 > >> sysctl -w net.link.bridge.pfil_local_phys=3D0 > >> Nothing has changed for the linux VM. > >> Windows11 VM is getting download/upload speed of 40Mbps/37Mbps while a > Debian12 VM is getting download/upload of 37Mbps/45Kbps. > >> Maybe there is an issue with the Linux itself? > > > > I never had this solved. Even my FreeBSD guest on FreeBSD host with > VirtualBox is slow as few hunderds kB/s > > It was like 10Mbps with Bhyve. I only use VMs for testing but installin= g > packages is always so slow. So you are not alone. I would really like to > know how to improve the network speed with virtualized environment. > > This looks weird to me. I run lots of VMs in production on TrueNAS CORE > - essentially FreeBSD 13-STABLE with bhyve and all of them get near gigab= it > speed with bridged networking. > > Guests: > > Windows > Ubuntu > FreeBSD (OPNsense) > > Specifically the OPNsense VM can route 700-800 Mbit/s across gigabit > interfaces. > > All my VMs use VirtIO network drivers - do yours? > > Odhiambo, another minor thing: > > > ifconfig_em1=3D"inet w.x.y.z netmask 255.255.255.0 mtu 1492 -tso -lro > -txcsum -rxcsum" > > A bridge member interface must not have a layer 3 address. You must put > the IP address > on the bridge interface itself and only configure > > ifconfig_em1=3D"mtu 1492 -tso -lro -txcsum -rxcsum up" > em1 is NOT a bridge member. It's the host's interface that is connected to the Internet. So the suggestion by @Wanpeng was "you have to turn off TSO LRO TXCSUM etc for the host interface which is bridge to VM", which I understood as my main interface. My bridge configuration is as follows: cloned_interfaces=3D"bridge0 tap0 tap1 tap2 tap3" ifconfig_bridge0_name=3D"em1bridge" ifconfig_em1bridge=3D"addm em1 addm tap0 addm tap1 addm tap2 addm tap3 up" ifconfig_tap0=3D"inet 172.16.1.1/24" ifconfig_tap1=3D"inet 172.16.2.1/24" ifconfig_tap2=3D"inet 172.16.3.1/24" ifconfig_tap3=3D"inet 172.16.4.1/24" When I create a VM and tie it on tap0, I give the VM an IP like 172.16.1.10/24, with a gateway of 172.16.1.1. But this is most probably not connected to your performance problem. It > just breaks > other things if you have an IP address on a bridge member ;-) > So is the setup above problematic or I am on the right track? > Why are you lowering the MTU of em1? I don't quite remember why I did that as it's a relic from the past :) > Does that carry over to the bridge interface? > Of course, because em1 is a bridge member, no? > What's the MTU of the emulated interface in your Linux VM? > Removing the MTU change on em1 has resolved the problem. The Debian12 VM now gets download/upload of 45Mbps/39Mbps! One last question for today (although I should just go and RTFM): Do I really need several tap devices? Can't I just have all my VMs on tap0? Each with it's own IP in that range? --=20 Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", egrep -v '^$|^.*#' =C2=AF\_(=E3=83=84)_/=C2=AF :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html] --000000000000c7b1450602e06b7e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Mon, Aug 14, 2023 at 12:38=E2=80= =AFPM Patrick M. Hausen <hausen@punkt.de> wrote:
Hi all,

> Am 14.08.2023 um 11:30 schrieb Miroslav Lachman <000.fbsd@quip.cz>:
>
> On 14/08/2023 10:49, Odhiambo Washington wrote:
>> I had the following in sysctl.conf:
>> net.link.tap.up_on_open=3D1
>> net.link.bridge.pfil_member=3D0
>> net.link.bridge.pfil_bridge=3D0
>> net.link.bridge.pfil_local_phys=3D1
>> So I only did:
>> sysctl -w net.link.bridge.pfil_onlyip=3D0
>> sysctl -w net.link.bridge.pfil_local_phys=3D0
>> Nothing has changed for the linux VM.
>> Windows11 VM is getting download/upload speed of 40Mbps/37Mbps whi= le a Debian12 VM is getting download/upload of 37Mbps/45Kbps.
>> Maybe there is an issue with the Linux itself?
>
> I never had this solved. Even my FreeBSD guest on FreeBSD host with Vi= rtualBox is slow as few hunderds kB/s
> It was like 10Mbps with Bhyve. I only use VMs for testing but installi= ng packages is always so slow. So you are not alone. I would really like to= know how to improve the network speed with virtualized environment.

This looks weird to me. I run lots of VMs in production on TrueNAS CORE
- essentially FreeBSD 13-STABLE with bhyve and all of them get near gigabit=
speed with bridged networking.

Guests:

Windows
Ubuntu
FreeBSD (OPNsense)

Specifically the OPNsense VM can route 700-800 Mbit/s across gigabit interf= aces.

All my VMs use VirtIO network drivers - do yours?

Odhiambo, another minor thing:

> ifconfig_em1=3D"inet w.x.y.z netmask 255.255.255.0 mtu 1492 -tso = -lro -txcsum -rxcsum"

A bridge member interface must not have a layer 3 address. You must put the= IP address
on the bridge interface itself and only configure

ifconfig_em1=3D"mtu 1492 -tso -lro -txcsum -rxcsum up"

em1 is NOT a bridge member. It's the host'= ;s interface that is connected to the Internet.
So the suggestion= by=C2=A0=C2=A0@Wanpeng was "you have to turn off TSO LRO TXCSUM etc f= or the host interface which is bridge to VM", which I understood
=
as my main interface.
My bridge configuration is as follows:=
cloned_interfaces=3D"bridge0 tap0 tap1 tap2 tap3"
i= fconfig_bridge0_name=3D"em1bridge"
ifconfig_em1bridge=3D"= addm em1 addm tap0 addm tap1 addm tap2 addm tap3 up"
ifconfig_tap0= =3D"inet 172.16.1.1= /24"
ifconfig_tap1=3D"inet 172.16.2.1/24"
ifconfig_tap2=3D"inet <= a href=3D"http://172.16.3.1/24" target=3D"_blank">172.16.3.1/24"ifconfig_tap3=3D"inet 172.16.4.1/24"

When I create a V= M and tie it on tap0, I give the VM an IP like 172.16.1.10/24, with a gateway of 172.16.1.1.

But this is most probably not connected to your performance problem. It jus= t breaks
other things if you have an IP address on a bridge member ;-)

So is the setup above problematic or I am on the rig= ht track?
=C2=A0
Why are you lowering the MTU of em1?
I don't quite re= member why I did that as it's a relic from the past :)
=C2=A0=
Does that carry ove= r to the bridge interface?

Of course, b= ecause em1 is a bridge member, no?
=C2=A0
What's the MTU of the emulated interface in your Linux VM?

Removing the MTU change on em1 has resolved the pro= blem. The Debian12 VM now gets download/upload of 45Mbps/39Mbps!
=
=C2=A0One last question for today (although I should just go= and RTFM): Do I really need several tap devices? Can't I just have all= my VMs on tap0? Each with it's own IP in that range?
<= br>
--
Best r= egards,
Odhiambo WASHINGTON,
Nairobi,KE
+254 7 3200 0004/+254 7 22= 74 3223
"Oh, the cruft.",=C2=A0egrep -v '^$|^.*#'=C2=A0=C2=AF\_(= =E3=83=84)_/=C2=AF=C2=A0:-)<= /div>
[How to ask smart questions:=C2= =A0http://www.catb.org/~esr/f= aqs/smart-questions.html]
--000000000000c7b1450602e06b7e--