ix0 and ix1 ifconfig options different on Supermicro board

Wed Nov 27 17:50:19 UTC 2019

Thank you, Vincenzo.

To anyone else looking this up, 'iperf' is not dependent on the TSO/LRO.
   iperf -c 10.1.1.1 -P 4   --->  9.4Gbps
   iperf3 -c 10.1.1.1 -P 4  --->  5.1Gbps

I still find it odd that the default FreeBSD install has different 
options for ix0 and ix1.

Rudy

On 11/23/19 12:51 AM, Vincenzo Maffione wrote:
> Hi,
>    TSO/LRO (for IPv4 and/or IPv6) will increase TCP bulk throughput on
> machine X for those TCP connection where X is one of the two endpoints,
> that is TCP connections that are local to X. That's why you are seing iperf
> achieving higher throughput with TSO/LRO enabled.
> TSO means that your local TCP stack will pass down large (e.g. 32K) packets
> to the NIC driver, and the NIC will take care of segmentation. This is
> beneficial for two reasons: (1) the segmentation work is done in hardware
> rather than in the CPU, and this is typically faster (and also, you save
> the CPU time for other stuff); (2) the per-packet cost of protocol
> processing (TCP, IP, Ethernet) is amortized over a large amount of bytes,
> which means that your total per-byte CPU time will be way lower. Most of
> the gain actually comes from (2).
> LRO is similar, but in the receive direction.
>
> However, if your device is a router it means that it forwards packets.
> Therefore the local TCP stack is not involved, so TSO simply does not apply
> (at least in FreeBSD).
> I think LRO applies, but there is a latency hit, as suggested by the wiki
> page you pointed.
>
> So no, enabling TSO/LRO will not increase the forwarding rate, but possibly
> increase latency. You should keep it disabled.
>
> Cheers,
>    Vincenzo
>
> Il giorno ven 22 nov 2019 alle ore 22:47 BulkMailForRudy <
> crapsh at monkeybrains.net> ha scritto:
>
>> I just did another test to a machine with a Chelsio card.
>>
>>    Server D (cxl3) -> Server A = 3.5Gbps
>>
>> Turning on flags lro tso4 tso6 vlanhwtso , yields
>>
>>    Server D (cxl3) -> Server A = 9.1 Gbps
>>
>> Oddly, this was an ipv4 iperf, but tso6 seems to help.
>>
>> I had settings turned off per
>> https://wiki.freebsd.org/10gFreeBSD/Router#Disabling_LRO_and_TSO
>>
>> Servers A,B, and C are all running services.  Server D is acting as a
>> router.  Are the LRO and TSO only for TCP to the box, or will it
>> increase speeds for forwarding if I enable it?
>>
>>
>> Thanks,
>>
>> Rudy
>>
>>
>> On 11/22/19 1:30 PM, BulkMailForRudy wrote:
>>> I have nearly identical setups, but ix0 and ix1 are getting different
>>> options at boot.  This seems to be the only difference I see between
>>> machines and I am trying to answer the question, Why can Server A
>>> iperf close to line rate while the other servers can not?
>>>
>>> The Test:  iperf -P 3 -c REMOTE_ADDR
>>>
>>> Server A (ix1) -> Server C (ix0)  = 9.4Gbps
>>> Server B (ix0)-> Server C (ix0) = 5.6Gbps
>>> Server C (ix0)-> A (ix1) or B (ix0)  = 5.0Gbps
>>>
>>>
>>> The motherboards are identical between A,B and C and the configs very
>>> similar.  The only difference is that Server A is plugged into ix1
>>> while Server B and C are using ix0.
>>>
>>>
>>> I am not modifying the flags at boot (eg ifconfig -tso), yet ix0 lacks
>>> TXCSUM,TSO4,TSO6,LRO,WOL.
>>>
>>> ix0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric
>>> 0 mtu 1500
>>>
>> options=a538b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6>
>>
>>>          ether *ac:1f:6b:6a:14:6*4
>>>          media: Ethernet autoselect (10Gbase-T <full-duplex>)
>>> ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>>>
>> options=e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
>>
>>>          ether *ac:1f:6b:6a:14:6*5
>>>          media: Ethernet autoselect (10Gbase-T <full-duplex>)
>>>
>>> I did try adding some flags to ix0 and -- not sure if this was the
>>> reason -- the box started acting oddly and I ended up rebooting it.
>>>
>>>
>>> My hunch has is that there is somethign with the TSO4.
>>>
>>>
>>> Rudy
>>>
>>> _______________________________________________
>>> freebsd-net at freebsd.org mailing list
>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>>>
>> _______________________________________________
>> freebsd-net at freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>>
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>