Re: 25/100 G performance on freebsd

Hi Benoit,

Not sure what the environment, is this to host VNF? those 2x25 will be 
both forwardings or are active/standby).

In my case I use:

    *    Vale for Inter-VM inside the same host.

    *    Vale to connect to the external network ( hence a phy 
interface). In my case Intel 40G NICS.

    *    SR-IOV for some specific use cases (for example, BNG stress 
test tools running on Linux).


For JAILS:

     *    I tend to use just VNET. Can't get more than 7.2Gbps ( >1400b) 
from an epair without a bridge in the middle.

     *    Right now I'm doing some tests with RSS enabled, but is not 
looking good, actually no passing traffic...


If your NICs start to play nice with SR-IOV you can pass a VF to the 
Jail, some NICs allow creating L2 "high speed" switches in the card ( 
never used one).

Regarding L3 (in-kernel), the overhead will be bigger than using vale, 
but then you can leverage multi-path, VXLAN termination, IPFW, PF, 
dummynet,  etc.

Hope it makes sense.

Santi


On 8/13/22 11:20, Benoit Chesneau wrote:
>
> Santiago thanks for the help.
>
> I am curious about your vale setup. Do you have only internal bridges? 
> Do you bridge the NIC interface or are doing L3?
>
> Afaik i am trying to dind what would be the most efficient way to use 
> the 25GB interfaces whle isolating the services on them.  I very 
> hesitant of the approach and unsure if freebsd these days can fit the 
> bill:
>
> * run isolated services over the 2x25G . would jails limit the bandwith?
> * possibly run bhyve services when linux or else  is needed .
>
> Would using only L3 routing solve some performances issues?
>
>
> benoit
>
>
> On Wed, Aug 10, 2022 at 23:31, Santiago Martinez <sm@codenetworks.net> 
> wrote:
>> Hi Benoit, sorry to hear that the SR-IOV still not working on your HW.
>>
>> Have you tested the last patch from Intel?
>>
>> Regarding Bhyve, you can use Vale switches (based on netmap).
>> On my machines, i get around ~33Gbps between VM (same local machine), 
>> sometimes going towards 40Gbps... ( These are basic tests with iperf3 
>> and TSO/LRO enabled).
>>
>> @Michael Dexter is working on a document that contains configuration 
>> examples and test results for the different network backend available 
>> in bhyve.
>>
>> If you need help, let me know and we can set up a call.
>> Take care.
>> Santi
>>
>> On 8/8/22 08:57, Benoit Chesneau wrote:
>>> For some reasons. I can’t use SR-IOV on my freebsd machines (HPE 
>>> DL160 gen10) with latest 25G HPE branded cards. I opened tickets for 
>>> that but since then no move happened.
>>>
>>> So I wonder id there is a good setup to use these cards with the 
>>> virtualization. Which kind of performance should I expect using 
>>> if_bridge? What if i am doing L3 routing instead using epair or tap 
>>> (for bhyve). Would it work better?
>>>
>>> Any hint is welcome,
>>>
>>> Benoît
>>>