Chelsio netmap support ? (RELENG_11)

[Mike Tancsa]

On 3/7/2017 5:07 PM, Navdeep Parhar wrote:
> On Tue, Mar 7, 2017 at 1:53 PM, Mike Tancsa <mike at sentex.net> wrote:
> ...
>>
>> Using netsend, I cant seem to blast through a single flow of packets
>> greater than 800Kpps without packet loss.  Can you point me to any
>> performance tweaks for forwarding / routing ?
>>
>> I have 3 boxes, with one in the middle
>>
>> (netsend box) <----> (router with chelsio nics) <---> (netreceive box)
> 
> How is the router configured -- is this something netmap based?
> Please provide more details of the configuration.  What kind of
> CPU/chipset is it?

Hi,
 Xeon E3-1225 v5 @ 3.30GHz
 Supermicro X11SSL-F Intel Skylake-S/Skylake-H/Greenlow
 PCI-e 3.0 slots

t5iov0 at pci0:2:0:0:      class=0x020000 card=0x00001425 chip=0x50071425
rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T520-SO Unified Wire Ethernet Controller'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 64, base 0xdd800000, size 524288,
enabled
    bar   [18] = type Memory, range 64, base 0xdd780000, size 524288,
enabled
    bar   [20] = type Memory, range 64, base 0xdd88c000, size 8192, enabled
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 8 messages, 64 bit, vector masks
    cap 10[70] = PCI-Express 2 endpoint max data 256(2048) FLR NS
                 link x8(x8) speed 8.0(8.0)
    cap 11[b0] = MSI-X supports 34 messages
                 Table in map 0x20[0x0], PBA in map 0x20[0x1000]
    cap 03[d0] = VPD
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
    ecap 0002[140] = VC 1 max VC1
    ecap 0003[170] = Serial 1 0000000000000000
    ecap 000e[190] = ARI 1
    ecap 0019[1a0] = PCIe Sec 1 lane errors 0
    ecap 0010[1c0] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI
disabled
                     0 VFs configured out of 16 supported
                     First VF RID Offset 0x0008, VF RID Stride 0x0004
                     VF Device ID 0x5807
                     Page Sizes: 4096 (enabled), 8192, 65536, 262144,
1048576, 4194304


Two 10G interfaces on a switch vcxl0 (192.168.1.1) vcxl1 (10.151.10.1)

192.168.1.2 does the netsend to 10.151.10.3 so it goes through the
router in question.  Router in question is RELENG11

dev.cxl.0.pause_settings=0
dev.cxl.1.pause_settings=0

loader.conf
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
zfs_load="YES"
#t4fw_cfg_load="YES"
t5fw_cfg_load="YES"
#t6fw_cfg_load="YES"
if_cxgbe_load="YES"
hw.cxgbe.num_vis=2
comconsole_speed="115200"       # Set the current serial console speed
console="comconsole,vidconsole"         # A comma separated list of
console(s)
comconsole_port="0x2F8"


> Do you see any PAUSE frames out of the T5? "sysctl dev.cxl dev.vcxl |
> grep _pause"  Is any CPU core pegged at 100% during the test?


dev.cxl.1.stats.rx_pause: 0
dev.cxl.1.stats.tx_pause: 0
dev.cxl.1.pause_settings: 0
dev.cxl.0.stats.rx_pause: 0
dev.cxl.0.stats.tx_pause: 0
dev.cxl.0.pause_settings: 0


while forwarding, top shows the system 25% in interrupt

# dmesg | grep netm
netmap: loaded module
vcxl0: netmap queues/slots: TX 2/1023, RX 2/1024
vcxl0: 1 txq, 1 rxq (NIC); 1 txq, 1 rxq (TOE); 2 txq, 2 rxq (netmap)
vcxl1: netmap queues/slots: TX 2/1023, RX 2/1024
vcxl1: 1 txq, 1 rxq (NIC); 1 txq, 1 rxq (TOE); 2 txq, 2 rxq (netmap)
igb0: netmap queues/slots: TX 4/1024, RX 4/1024
igb1: netmap queues/slots: TX 4/1024, RX 4/1024

It maxes out at about 800Kpps with and without netmap.  Is there a way
to increase the queues for the Chelsio nic, like the onboard igb ?

 top -nCHSIzs1
last pid:  2517;  load averages:  1.15,  0.45,  0.27  up 0+04:26:13
20:44:09
393 processes: 8 running, 346 sleeping, 39 waiting

Mem: 10M Active, 37M Inact, 360M Wired, 15G Free
ARC: 67M Total, 691K MFU, 64M MRU, 16K Anon, 430K Header, 1758K Other
Swap: 32G Total, 32G Free


  PID USERNAME   PRI NICE   SIZE    RES STATE   C   TIME     CPU COMMAND
   11 root       -92    -     0K   672K CPU0    0   5:45  80.08%
intr{irq272: t5nex0:0b0}



	---Mike

-- 
-------------------
Mike Tancsa, tel +1 519 651 3400
Sentex Communications, mike at sentex.net
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada   http://www.tancsa.com/