10G performance regression / difference cxl and ix RELENG11 vs HEAD

[Mike Tancsa]

I was doing a quick iperf test with  r339328 GENERIC-NODEBUG  amd64, and
noticed  I can no longer saturate a 10G nic with iperf3.  I tried first
with the ix adapter, but was not sure if it was the driver or not. I
tried as well with a Chelsio and got similar numbers.

# iperf3 -c 192.168.242.3
Connecting to host 192.168.242.3, port 5201
[  5] local 192.168.242.2 port 50474 connected to 192.168.242.3 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   997 MBytes  8.36 Gbits/sec  717    175
KBytes      
[  5]   1.00-2.00   sec   975 MBytes  8.18 Gbits/sec  668   41.1
KBytes      
[  5]   2.00-3.00   sec   880 MBytes  7.38 Gbits/sec  846   25.6
KBytes      
[  5]   3.00-4.00   sec   523 MBytes  4.39 Gbits/sec  802   59.8
KBytes      
[  5]   4.00-5.00   sec   520 MBytes  4.36 Gbits/sec  882   48.4
KBytes      
[  5]   5.00-6.00   sec   543 MBytes  4.55 Gbits/sec  838   56.9
KBytes      
[  5]   6.00-7.00   sec   556 MBytes  4.66 Gbits/sec  850   11.4
KBytes      
[  5]   7.00-8.00   sec   538 MBytes  4.51 Gbits/sec  795   39.9
KBytes      
[  5]   8.00-9.00   sec   540 MBytes  4.53 Gbits/sec  853   57.0
KBytes      
[  5]   9.00-10.00  sec   503 MBytes  4.22 Gbits/sec  814   59.8
KBytes      
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  6.42 GBytes  5.52 Gbits/sec  8065            
sender
[  5]   0.00-10.00  sec  6.42 GBytes  5.52 Gbits/sec                 
receiver


If I do a parallel transfer I get closer to the max.  However, on
RELENG11 I could do pretty close to a full 10G no problem with just a
single stream. This is with a GENERIC-NODEBUG kernel using a couple of
T520s back to back


t5iov1 at pci0:9:0:1:      class=0x020000 card=0x00001425 chip=0x50011425
rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T520-CR Unified Wire Ethernet Controller'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 64, base 0xf6680000, size 524288,
enabled
    bar   [18] = type Memory, range 64, base 0xf6600000, size 524288,
enabled
    bar   [20] = type Memory, range 64, base 0xf688a000, size 8192, enabled
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 8 messages, 64 bit, vector masks
    cap 10[70] = PCI-Express 2 endpoint max data 512(2048) FLR RO NS
                 link x8(x8) speed 8.0(8.0)



-- 
-------------------
Mike Tancsa, tel +1 519 651 3400 x203
Sentex Communications, mike at sentex.net
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada