[Bug 248652] iflib: netmap pkt-gen large TX performance difference between 11-STABLE and 12-STABLE/CURRENT on ix & ixl NIC

[bugzilla-noreply at freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248652

--- Comment #26 from Sylvain Galliano <sg at efficientip.com> ---
(In reply to Vincenzo Maffione from comment #25)

you are correct, using 1 queue on both ix/ixl NIC + tuning new sysctl, we have
same result as FreeBSD 11

Regarding avg_batch values on ixl, they are very low, not in the range you
expected:

6 queues, iflib.nm_tx_tmr_us=5
070.664142 main_thread [2639] 13.588 Mpps (13.602 Mpkts 6.522 Gbps in 1001000
usec) 20.87 avg_batch 99999 min_space
cpu usage: 100%

even with 1 queue I got a low value for avg_batch:
283.855379 main_thread [2639] 12.147 Mpps (12.757 Mpkts 5.831 Gbps in 1050245
usec) 13.97 avg_batch 99999 min_space
cpu usage: 100%

ix (X520) NIC have a good avg_batch (whatever the number of queue):
404.130120 main_thread [2639] 14.880 Mpps (14.895 Mpkts 7.143 Gbps in 1000999
usec) 436.04 avg_batch 99999 min_space

I don't known if this can help you but I did another test specific to ixl NIC:
setting hw.ixl.enable_head_writeback=0

avg_batch is higher, pps a little better and we are not CPU bound this time:

6 queues:
603.651106 main_thread [2639] 17.384 Mpps (17.402 Mpkts 8.345 Gbps in 1001003
usec) 308.10 avg_batch 99999 min_space
cpu usage: 71%

1 queue:
730.590104 main_thread [2639] 15.084 Mpps (15.416 Mpkts 7.241 Gbps in 1022004
usec) 442.64 avg_batch 99999 min_space
cpu usage: 57%


Same test but using more threads on pkt-gen (-p 6), 6 queues:
995.887010 main_thread [2639] 17.327 Mpps (17.339 Mpkts 8.317 Gbps in 1000693
usec) 286.35 avg_batch 599994 min_space
cpu usage: 197%

top -H:
  PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
70876 root        -92    0   348M    16M select   3   0:23  35.46%
pkt-gen{pkt-gen}
70876 root        -92    0   348M    16M CPU4     4   0:23  33.58%
pkt-gen{pkt-gen}
70876 root        -92    0   348M    16M RUN      6   0:23  33.44%
pkt-gen{pkt-gen}
70876 root        -92    0   348M    16M select   2   0:23  31.69%
pkt-gen{pkt-gen}
70876 root         45    0   348M    16M CPU9     9   0:23  31.64%
pkt-gen{pkt-gen}
70876 root        -92    0   348M    16M select   6   0:23  31.24%
pkt-gen{pkt-gen}

-- 
You are receiving this mail because:
You are on the CC list for the bug.