From nobody Wed Dec 01 22:15:58 2021 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id E59BF18C1C66 for ; Wed, 1 Dec 2021 22:16:04 +0000 (UTC) (envelope-from SRS0=qlzq=QS=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4J4D1w5NLNz3r7Y; Wed, 1 Dec 2021 22:16:04 +0000 (UTC) (envelope-from SRS0=qlzq=QS=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 0B01A28411; Wed, 1 Dec 2021 23:16:03 +0100 (CET) Received: from illbsd.quip.test (ip-78-45-215-131.net.upcbroadband.cz [78.45.215.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id E75FF28417; Wed, 1 Dec 2021 23:15:59 +0100 (CET) Subject: Re: bhyve vCPU limit To: Oleg Ginzburg , rgrimes@freebsd.org Cc: bsdlists@jld3.net, jbo@insane.engineer, freebsd-virtualization@freebsd.org References: <4E8A7FD3-B01E-4ADE-A290-360F3B04AC0F@jld3.net> <30e4454c-414a-833f-3829-586a450e7205@quip.cz> From: Miroslav Lachman <000.fbsd@quip.cz> Message-ID: <8a27cc73-4c68-a643-5809-d6ec87201eb2@quip.cz> Date: Wed, 1 Dec 2021 23:15:58 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4J4D1w5NLNz3r7Y X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N On 01/12/2021 19:13, Oleg Ginzburg wrote: > Hi > > On Wed, Dec 1, 2021 at 7:52 PM Miroslav Lachman <000.fbsd@quip.cz> wrote: >> I was playing with VMs in VirtualBox and Bhyve and compared >> performance with increasing vCPU count. The more cores VM get the slower > > How many physical cores you have on the host? This is a characteristic > behavior when overcoming. > But without overcommit I observe productivity growth: I tested it on 2 machines. One had 6 cores 12 threads, the second 4 cores 8 threads. Even if the host machine was almost idle, only one VM was running the slowness with more than 1 vCPU was so significant. Let's say 0.01 sec for 1 vCPU, 0.3 s for 2 vCPU, 0.9 s for 4 vCPU. (on machine with real 4 cores / 8 threads). With overcommit of vCPUs it was like 3 seconds with 6 vCPU and 8 seconds for 8 vCPU. Still the same task on the same idle machine. On the other hand I can start 5 VMs with 2 vCPUs each and "everything seems normal" even if the total count of vCPUs are 10 on machine with 4 physical / 8 threads. In fact I tested with total count of 15 vCPUs distributed between 5+ VMs but the performance problem was always visible on VM with more than 2 vCPU. > 1) > Single vCore benchmark via cpuminer: https://pastebin.com/mg46RvDT > TOP from host: https://pastebin.com/LvbBiyFz > > > 2) > Sixteen vCore benchmark via cpuminer: https://pastebin.com/7FQjaVM8 > TOP from host: https://pastebin.com/dLpreq9D > > I observe almost linear growth. > > In addition, the loss of performance can be associated with NUMA > (required cpuset/cpuset_setaffinity > and control of memory capacity) > > As far as I remember, it is not enough to simply change one parameter > in .h file: https://bhyvecon.org/bhyveconOttawa2019-Rodney.pdf Thank you for your data! Kind regards Miroslav Lachman