Re: Comments on the latest phoronix benchmark

I've been working on a neutral numerical  benchmarking interface that uses
Python/R. I haven't thought of using it to compare different versions of
FreeBSD. I wrote it to compare FreeBSD/Linux for compute intensive
workflows. But it would be interesting to compare different versions. I
rarely expect the differences between OS versions to be that different, but
perhaps that's a mistake.

Folks at Klara (that do a lot of impressive benchmarking with FreeBSD)
always recommend writing your own tests to capture performance data. Your
workloads could be web traffic, databases, compute, etc... but it very
likely will not look much like the pre-canned tests that Phoronix puts
together. I do look at Larabel's test results on BSD. I follow it with
interest, even. But I tend to treat it as early warning testing. I'm
usually left trying to figure out a poor result for FreeBSD on my own. In
my own CPU/RAM/Filesystem benchmarking workflows on standard installations,
FreeBSD and Linux are often very close.

Best - Kyle


[1]
https://klarasystems.com/articles/evaluating-freebsd-current-for-production-use/
[2] https://the-integral.dev/post/freebsd-for-data-science/



On Tue, Jun 18, 2024, 3:09 PM Warner Losh <imp@bsdimp.com> wrote:

>
>
> On Tue, Jun 18, 2024, 2:08 PM Yonas Yanfa <yonas.yanfa@gmail.com> wrote:
>
>> Hi,
>>
>> Link: https://www.phoronix.com/review/bsd-linux-threadripper-7980x/3
>>
>> The following six benchmarks show FreeBSD 14.1 performed poorly compared
>> to other OSes:
>>
>> [ 4.3x slower ] Stress-NG 0.17.08: Socket Activity -- 3,569 (FreeBSD) vs
>> 15,267 (CentOS Stream 9)
>> [ 2.9x slower ] Stress-NG 0.17.08: Glibc Qsort Data Sorting -- 779
>> (FreeBSD) vs 2,224 (Ubuntu 24.04 LTS)
>> [ 2.2x slower ] Stress-NG 0.17.08: AVX-512 VNNI -- 3,626,943 (FreeBSD)
>> vs 8,253,203 (Ubuntu 24.04 LTS)
>> [ 1.5x slower ] Stress-NG 0.17.08: CPU Cache -- 2,322,478 (FreeBSD) vs
>> 3,557,329 (NetBSD)
>> [ 1.5x slower ] Stress-NG 0.17.08: Fused Multiply-Add -- 63,639,465
>> (FreeBSD) vs 96,258,730 (Ubuntu 24.04 LTS)
>> [ 1.3x slower ] Stress-NG 0.17.08: Semaphores -- 230,741,240 (FreeBSD)
>> vs 313,648,228 (DragonFlyBSD)
>>
>> Does anyone know why, and how we can improve the numbers?
>>
>
> Stress-ng is not intended to be a benchmark (and says so in its docs) and
> does all kinds of special things on Linux only. It has a bunch of stubs on
> systems that didn’t implement something. It's a deeply flawed. I believe
> this information is in the comments to the article.
>
> That said, there are speed improvements we can make to things, like our VM
> that other benchmarks do show issues with... but first the benchmarks need
> to actually be apples to apples comparisons.
>
> Warner
>
> Cheers,
>> Yonas
>>
>>
>>