From nobody Wed Dec 01 22:01:04 2021 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 6FFCB18BBF8D for ; Wed, 1 Dec 2021 22:01:12 +0000 (UTC) (envelope-from SRS0=qlzq=QS=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4J4Chm1kkJz3mJL for ; Wed, 1 Dec 2021 22:01:12 +0000 (UTC) (envelope-from SRS0=qlzq=QS=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 21D0C28417; Wed, 1 Dec 2021 23:01:10 +0100 (CET) Received: from illbsd.quip.test (ip-78-45-215-131.net.upcbroadband.cz [78.45.215.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id B9BE828411; Wed, 1 Dec 2021 23:01:05 +0100 (CET) Subject: Re: bhyve vCPU limit To: Paul Vixie , freebsd-virtualization@freebsd.org Cc: bsdlists@jld3.net, jbo@insane.engineer References: <4E8A7FD3-B01E-4ADE-A290-360F3B04AC0F@jld3.net> <30e4454c-414a-833f-3829-586a450e7205@quip.cz> <83a44a1b-c7a8-31e6-e212-7a9a04af3fb2@redbarn.org> From: Miroslav Lachman <000.fbsd@quip.cz> Message-ID: <99d2ceb8-94d3-2115-3ac5-1791b5145378@quip.cz> Date: Wed, 1 Dec 2021 23:01:04 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 In-Reply-To: <83a44a1b-c7a8-31e6-e212-7a9a04af3fb2@redbarn.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4J4Chm1kkJz3mJL X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N On 01/12/2021 19:13, Paul Vixie wrote: > > > Miroslav Lachman wrote on 2021-12-01 08:52: >> On 01/12/2021 17:17, John Doherty via freebsd-virtualization wrote: >>> ... >> >> I am sorry for hijacking this thread but your information is very >> interesting. I was playing with VMs in VirtualBox and Bhyve and >> compared performance with increasing vCPU count. The more cores VM get >> the slower was even a simple single threaded task like loading PF >> rules from /etc/pf.conf. It was tested on FreeBSD 11.4 and 12.2, I >> tested ULE and 4BSD schedulers. Maybe it was somewhat HW related but >> it always shows VMs with more than 2 v CPUs significantly slower. VMs >> with 6+ vCPU was almost unusable (loading of PF ruleset takes about 8 >> seconds instead of fraction on single vCPU VM). >> >> ... > > loading a PF ruleset requires a fair bit of locking and unlocking of > kernel data structures for each system call, per rule. while pfctl is > single threaded, the acquisition process of those kernel locks probably > requires a memory buffer flush to guaranty atomicity, and the lock's > domain may overlap with other non-PF kernel activities that different > hypervisors virtualize differently. > > this makes loading a PF ruleset a poor benchmark for hypervisors, unless > that activity is so common that the unusable slowness is interfering > with other work. it could be debugged or optimized in that case, but how > often do you really need to add a PF ruleset? I don't take pfctl as a benchmark. The whole case started with bad webserver performance (Apache + PHP) so I added 2 more vCPUs and the problem was even bigger - performance slower. It was so slow that on reboot I think it freezed on loading PF rules. That's why I take it as an example because it was the most visible. Loading of rules was not the only one problem. I can live with 10 seconds of loading PF rules but not with the bad webserver performance. This is an old case which resulted in migration of client to another service provider which uses jails instead of bhyve = no performance problems with similar HW setup. Miroslav Lachman