Re: bhyve vCPU limit

On Wed 2021-12-01 09:52 AM MST -0700, <000.fbsd@quip.cz> wrote:

> On 01/12/2021 17:17, John Doherty via freebsd-virtualization wrote:
>> That limitation appears to still exist in FreeBSD 13.0-RELEASE:
>>
>> [root@grit] # freebsd-version -k ; grep 'VM_MAXCPU' 
>> /usr/src/sys/amd64/include/vmm.h
>> 13.0-RELEASE
>> #define    VM_MAXCPU    16            /* maximum 
>> virtual cpus */
>>
>> I ran into this in May 2021 and with some help from folks on this 
>> list was able to increase it. The simplest (if not minimalist) way to 
>> do that is:
>>
>> 1. edit /usr/src/sys/amd64/include/vmm.h to increase that value: I 
>> used 48
>> 2. make buildworld
>> 3. make installworld
>>
>> The increased value has been working fine for me since I did that. I 
>> run a couple of VMs with 24 vCPUs each and several others with 
>> smaller numbers all the time and have run others with as many as 48 
>> temporarily. No problems that I have seen.
>
> I am sorry for hijacking this thread but your information is very 
> interesting. I was playing with VMs in VirtualBox and Bhyve and 
> compared performance with increasing vCPU count. The more cores VM get 
> the slower was even a simple single threaded task like loading PF 
> rules from /etc/pf.conf. It was tested on FreeBSD 11.4 and 12.2, I 
> tested ULE and 4BSD schedulers. Maybe it was somewhat HW related but 
> it always shows VMs with more than 2 v CPUs significantly slower. VMs 
> with 6+ vCPU was almost unusable (loading of PF ruleset takes about 8 
> seconds instead of fraction on single vCPU VM).
>
> Do you have any special tunning to have so large number of vCPU 
> without this penalty?

I did not do anything special other than the steps described above. I 
did do some other stuff while sort of stumbling toward the eventual 
solution, but that's neither here nor there anymore. The steps above are 
what I used to build the primary system where I use bhyve. The physical 
host has two Xeon E5-2690 v4 CPUs, 14 cores/28 threads each so 28 
cores/56 threads total.

I have not seen but neither have I tried to measure any problems like 
you describe. bhyve works very well for me and I especially like it in 
combination with the vm-bhyve package, which I'm using to manage the 
VMs.

Of the various virtualization systems I've used or tried over the years, 
I like this combination more than any other. It's simple, clean, 
integrates well with ZFS, and is a pleasure to use, at least by my 
lights. Haven't had any trouble either before or after increasing the 
VM_MAXCPU value.