From nobody Thu Jun 30 10:45:54 2022 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id ABC9387EB74 for ; Thu, 30 Jun 2022 10:45:56 +0000 (UTC) (envelope-from nbe@vkf-renzel.de) Received: from mx1.renzel.net (mx1.renzel.net [195.243.213.130]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits)) (Client CN "*.renzel.net", Issuer "DigiCert TLS RSA SHA256 2020 CA1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4LYZkC6zlQz4VHJ for ; Thu, 30 Jun 2022 10:45:55 +0000 (UTC) (envelope-from nbe@vkf-renzel.de) Received: from [192.168.255.13] (unknown [94.31.91.26]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by mx1.renzel.net (smtpd) with ESMTPSA id E7A8DD75A2 for ; Thu, 30 Jun 2022 12:45:54 +0200 (CEST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.104.2 at clamav-milter.renzel.net Message-ID: Date: Thu, 30 Jun 2022 12:45:54 +0200 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.9.1 From: Nils Beyer Subject: bhyve: slow network throughput between guest VM and host (and vice versa)? To: freebsd-virtualization@FreeBSD.org Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4LYZkC6zlQz4VHJ X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of nbe@vkf-renzel.de designates 195.243.213.130 as permitted sender) smtp.mailfrom=nbe@vkf-renzel.de X-Spamd-Result: default: False [-2.50 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; RWL_MAILSPIKE_GOOD(0.00)[195.243.213.130:from]; R_SPF_ALLOW(-0.20)[+mx:c]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000]; RECEIVED_SPAMHAUS_PBL(0.00)[94.31.91.26:received]; DMARC_NA(0.00)[vkf-renzel.de]; RCVD_IN_DNSWL_MED(-0.20)[195.243.213.130:from]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; MLMMJ_DEST(0.00)[freebsd-virtualization]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:3320, ipnet:195.243.0.0/16, country:DE]; SUBJECT_ENDS_QUESTION(1.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N Hi, I've setup a FreeBSD VM and am using a VirtIO network interface within. Then I've setup an ip address 192.168.0.2/30 to that VirtIO NIC of the guest using: ifconfig vtnet0 192.168.0.2/30 up and on the TAP interface of the host an ip address 192.168.0.1/30 using ifconfig tap0 192.168.0.1/30 up Trying an iperf3-transfer (iperf3-server on the host, iperf3-client on the guest) with a TCP window size of 128k I only get around 2.45Gbit/s: > # env LD_LIBRARY_PATH=. ./iperf3 -c 192.168.0.1 -w 128k > Connecting to host 192.168.0.1, port 5201 > [ 5] local 192.168.0.2 port 25651 connected to 192.168.0.1 port 5201 > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.00 sec 314 MBytes 2.63 Gbits/sec 0 1.43 MBytes > [ 5] 1.00-2.00 sec 301 MBytes 2.52 Gbits/sec 0 1.43 MBytes > [ 5] 2.00-3.00 sec 264 MBytes 2.21 Gbits/sec 0 1.43 MBytes > [ 5] 3.00-4.00 sec 284 MBytes 2.38 Gbits/sec 0 1.43 MBytes > [ 5] 4.00-5.00 sec 296 MBytes 2.48 Gbits/sec 0 1.43 MBytes > [ 5] 5.00-6.00 sec 279 MBytes 2.34 Gbits/sec 0 1.43 MBytes > [ 5] 6.00-7.00 sec 280 MBytes 2.35 Gbits/sec 0 1.43 MBytes > [ 5] 7.00-8.00 sec 310 MBytes 2.60 Gbits/sec 0 1.43 MBytes > [ 5] 8.00-9.00 sec 302 MBytes 2.53 Gbits/sec 0 1.43 MBytes > [ 5] 9.00-10.00 sec 333 MBytes 2.79 Gbits/sec 0 1.43 MBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.00 sec 2.89 GBytes 2.49 Gbits/sec 0 sender > [ 5] 0.00-10.00 sec 2.89 GBytes 2.49 Gbits/sec receiver Switching the roles (iperf3-server on the guest, iperf3-client on the host) with a TCP windows size of 128k, I get 4.04Gbit/s: > #iperf3 -c 192.168.0.2 -w 128k > Connecting to host 192.168.0.2, port 5201 > [ 5] local 192.168.0.1 port 56892 connected to 192.168.0.2 port 5201 > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.00 sec 411 MBytes 3.44 Gbits/sec 40 973 KBytes > [ 5] 1.00-2.00 sec 483 MBytes 4.05 Gbits/sec 5 1.03 MBytes > [ 5] 2.00-3.00 sec 507 MBytes 4.26 Gbits/sec 1 1.21 MBytes > [ 5] 3.00-4.00 sec 514 MBytes 4.31 Gbits/sec 15 561 KBytes > [ 5] 4.00-5.00 sec 498 MBytes 4.18 Gbits/sec 10 966 KBytes > [ 5] 5.00-6.00 sec 491 MBytes 4.12 Gbits/sec 19 841 KBytes > [ 5] 6.00-7.00 sec 513 MBytes 4.31 Gbits/sec 0 1.43 MBytes > [ 5] 7.00-8.00 sec 504 MBytes 4.23 Gbits/sec 0 1.43 MBytes > [ 5] 8.00-9.00 sec 459 MBytes 3.85 Gbits/sec 0 1.43 MBytes > [ 5] 9.00-10.00 sec 435 MBytes 3.65 Gbits/sec 0 1.43 MBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.00 sec 4.70 GBytes 4.04 Gbits/sec 90 sender > [ 5] 0.00-10.00 sec 4.70 GBytes 4.04 Gbits/sec receiver Increasing MTU on the Virtio interface and on the TAP interface to 9000 helps a little bit: getting 8.38Gbit/s guest->host and 10.3Gbit/s host->guest. Increasing TCP windows size to 1024k only produces more retries and does nothing on the throughput. Is that expected that I'm not able to get more throughput within the bhyve network stack (guest <-> host)? I was expecting way more then 10Gbit/s... TIA and KR, Nils