Too aggressive TCP ACKs

From: Zhenlei Huang <zlei.huang_at_gmail.com>
Date: Fri, 21 Oct 2022 14:19:19 UTC
Hi,

While I was repeating https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=258755, I observed a
strange behavior. The TCP ACKs from FreeBSD host are too aggressive.

My setup is simple:
         A                                 B
   [ MacOS ]  <====> [ FreeBSD VM ]
192.168.120.1            192.168.12.134 (disable tso and lro)
While A <--- B, i.e. A as server and B as client, the packets rate looks good.

One session on B:

root@:~ # iperf3 -c 192.168.120.1 -b 10m
Connecting to host 192.168.120.1, port 5201
[  5] local 192.168.120.134 port 54459 connected to 192.168.120.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.25 MBytes  10.5 Mbits/sec    0    257 KBytes       
[  5]   1.00-2.00   sec  1.25 MBytes  10.5 Mbits/sec    0    257 KBytes       
[  5]   2.00-3.00   sec  1.12 MBytes  9.44 Mbits/sec    0    257 KBytes       
[  5]   3.00-4.00   sec  1.25 MBytes  10.5 Mbits/sec    0    257 KBytes       
[  5]   4.00-5.00   sec  1.12 MBytes  9.44 Mbits/sec    0    257 KBytes       
[  5]   5.00-6.00   sec  1.25 MBytes  10.5 Mbits/sec    0    257 KBytes       
[  5]   6.00-7.00   sec  1.12 MBytes  9.44 Mbits/sec    0    257 KBytes       
[  5]   7.00-8.00   sec  1.25 MBytes  10.5 Mbits/sec    0    257 KBytes       
[  5]   8.00-9.00   sec  1.12 MBytes  9.44 Mbits/sec    0    257 KBytes       
[  5]   9.00-10.00  sec  1.25 MBytes  10.5 Mbits/sec    0    257 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  12.0 MBytes  10.1 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  12.0 MBytes  10.1 Mbits/sec                  receiver

iperf Done.

Another session on B:

root@:~ # netstat -w 1 -I vmx0
            input           vmx0           output
   packets  errs idrops      bytes    packets  errs      bytes colls
         0     0     0          0          0     0          0     0
         0     0     0          0          0     0          0     0
       342     0     0      22600        526     0     775724     0
       150     0     0       9900        851     0    1281454     0
       109     0     0       7194        901     0    1357850     0
       126     0     0       8316        828     0    1246632     0
       122     0     0       8052        910     0    1370780     0
       109     0     0       7194        819     0    1233702     0
       120     0     0       7920        910     0    1370780     0
       110     0     0       7260        819     0    1233702     0
       123     0     0       8118        910     0    1370780     0
       109     0     0       7194        819     0    1233702     0
        73     0     0       5088        465     0     686342     0
         0     0     0          0          0     0          0     0
         0     0     0          0          0     0          0     0



================================================================


While A ---> B, i.e. A as client and B as server, the ACKs sent from B looks strange.

Session on A:

% iperf3 -c 192.168.120.134 -b 10m
Connecting to host 192.168.120.134, port 5201
[  5] local 192.168.120.1 port 52370 connected to 192.168.120.134 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.25 MBytes  10.5 Mbits/sec                  
[  5]   1.00-2.00   sec  1.25 MBytes  10.5 Mbits/sec                  
[  5]   2.00-3.00   sec  1.12 MBytes  9.44 Mbits/sec                  
[  5]   3.00-4.00   sec  1.25 MBytes  10.5 Mbits/sec                  
[  5]   4.00-5.00   sec  1.12 MBytes  9.44 Mbits/sec                  
[  5]   5.00-6.00   sec  1.25 MBytes  10.5 Mbits/sec                  
[  5]   6.00-7.00   sec  1.12 MBytes  9.44 Mbits/sec                  
[  5]   7.00-8.00   sec  1.25 MBytes  10.5 Mbits/sec                  
[  5]   8.00-9.00   sec  1.12 MBytes  9.44 Mbits/sec                  
[  5]   9.00-10.00  sec  1.25 MBytes  10.5 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  12.0 MBytes  10.1 Mbits/sec                  sender
[  5]   0.00-10.00  sec  12.0 MBytes  10.1 Mbits/sec                  receiver

iperf Done.

Session on B:

root@:~ # netstat -w 1 -I vmx0
            input           vmx0           output
   packets  errs idrops      bytes    packets  errs      bytes colls
         0     0     0          0          0     0          0     0
         0     0     0          0          0     0          0     0
       649     0     0     960562        330     0      21800     0
       819     0     0    1233702        415     0      27390     0
       910     0     0    1370780        459     0      30294     0
       819     0     0    1233702        415     0      27390     0
       910     0     0    1370780        459     0      30294     0
       910     0     0    1370780        460     0      30360     0
       819     0     0    1233702        414     0      27324     0
       910     0     0    1370780        460     0      30360     0
       819     0     0    1233702        414     0      27324     0
       910     0     0    1370780        460     0      30360     0
       285     0     0     412287        147     0       9981     0
         0     0     0          0          0     0          0     0
         0     0     0          0          0     0          0     0
         0     0     0          0          0     0          0     0


The ACK packets replied from B (the FreeBSD VM) are too aggressive. They are
about one half of TCP packets received from A.

I've tested with different bitrates, from 10m to 300m, all behave the same.
Tested with baremetal FreeBSD 13.1 Box as B (with intel em driver), the 
bitrates is 1g, also  behaves the same.

Also tried different FreeBSD versions, 11.4, 12.3, stable/13 and current/14 all 
behave the same.


My question is, is that the expected behavior of current default TCP stack?



Best regards,
Zhenlei