From nobody Wed Dec 01 16:52:19 2021 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 6B07E18C2B29 for ; Wed, 1 Dec 2021 16:52:33 +0000 (UTC) (envelope-from SRS0=qlzq=QS=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4J44rd1sB2z4Wy2 for ; Wed, 1 Dec 2021 16:52:33 +0000 (UTC) (envelope-from SRS0=qlzq=QS=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 1457128411; Wed, 1 Dec 2021 17:52:25 +0100 (CET) Received: from illbsd.quip.test (ip-78-45-215-131.net.upcbroadband.cz [78.45.215.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id CEC5F28417; Wed, 1 Dec 2021 17:52:20 +0100 (CET) Subject: Re: bhyve vCPU limit To: bsdlists@jld3.net, jbo@insane.engineer Cc: freebsd-virtualization@freebsd.org References: <4E8A7FD3-B01E-4ADE-A290-360F3B04AC0F@jld3.net> From: Miroslav Lachman <000.fbsd@quip.cz> Message-ID: <30e4454c-414a-833f-3829-586a450e7205@quip.cz> Date: Wed, 1 Dec 2021 17:52:19 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 In-Reply-To: <4E8A7FD3-B01E-4ADE-A290-360F3B04AC0F@jld3.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4J44rd1sB2z4Wy2 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N On 01/12/2021 17:17, John Doherty via freebsd-virtualization wrote: > That limitation appears to still exist in FreeBSD 13.0-RELEASE: > > [root@grit] # freebsd-version -k ; grep 'VM_MAXCPU' > /usr/src/sys/amd64/include/vmm.h > 13.0-RELEASE > #define    VM_MAXCPU    16            /* maximum virtual cpus */ > > I ran into this in May 2021 and with some help from folks on this list > was able to increase it. The simplest (if not minimalist) way to do that > is: > > 1. edit /usr/src/sys/amd64/include/vmm.h to increase that value: I used 48 > 2. make buildworld > 3. make installworld > > The increased value has been working fine for me since I did that. I run > a couple of VMs with 24 vCPUs each and several others with smaller > numbers all the time and have run others with as many as 48 temporarily. > No problems that I have seen. I am sorry for hijacking this thread but your information is very interesting. I was playing with VMs in VirtualBox and Bhyve and compared performance with increasing vCPU count. The more cores VM get the slower was even a simple single threaded task like loading PF rules from /etc/pf.conf. It was tested on FreeBSD 11.4 and 12.2, I tested ULE and 4BSD schedulers. Maybe it was somewhat HW related but it always shows VMs with more than 2 v CPUs significantly slower. VMs with 6+ vCPU was almost unusable (loading of PF ruleset takes about 8 seconds instead of fraction on single vCPU VM). Do you have any special tunning to have so large number of vCPU without this penalty? Kind regards Miroslav Lachman