ixl 40G bad performance?

[Eggert, Lars]

On 2015-10-26, at 18:40, Eggert, Lars <lars at netapp.com> wrote:
> On 2015-10-26, at 17:08, Pieper, Jeffrey E <jeffrey.e.pieper at intel.com> wrote:
>> As a caveat, this was using default netperf message sizes.
> 
> I get the same ~3 Gb/s with the default netperf sizes and driver 1.4.5.

Now there is version 1.4.8 on the Intel website, but it doesn't change things for me.

> When you tcpdump during the run, do you see TSO/LRO in effect, i.e., do you see "segments" > 32K in the trace?

I still see no TSO/LRO in effect when tcpdump'ing on the receiver; note how all the packets are 1448 bytes:

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ixl0, link-type EN10MB (Ethernet), capture size 262144 bytes
17:02:42.328782 IP 10.0.4.1.21507 > 10.0.4.2.12865: Flags [S], seq 15244366, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 478099 ecr 0], length 0
17:02:42.328808 IP 10.0.4.2.12865 > 10.0.4.1.21507: Flags [S.], seq 1819579546, ack 15244367, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 3553932482 ecr 478099], length 0
17:02:42.328842 IP 10.0.4.1.21507 > 10.0.4.2.12865: Flags [.], ack 1, win 1040, options [nop,nop,TS val 478099 ecr 3553932482], length 0
17:02:42.329804 IP 10.0.4.1.21507 > 10.0.4.2.12865: Flags [P.], seq 1:657, ack 1, win 1040, options [nop,nop,TS val 478100 ecr 3553932482], length 656
17:02:42.331671 IP 10.0.4.2.12865 > 10.0.4.1.21507: Flags [P.], seq 1:657, ack 657, win 1040, options [nop,nop,TS val 3553932485 ecr 478100], length 656
17:02:42.331717 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [S], seq 1387798477, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 478102 ecr 0], length 0
17:02:42.331729 IP 10.0.4.2.30216 > 10.0.4.1.10449: Flags [S.], seq 4085135109, ack 1387798478, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 2829000022 ecr 478102], length 0
17:02:42.331781 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], ack 1, win 1040, options [nop,nop,TS val 478102 ecr 2829000022], length 0
17:02:42.331796 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq 1:1449, ack 1, win 1040, options [nop,nop,TS val 478102 ecr 2829000022], length 1448
17:02:42.331800 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq 1449:2897, ack 1, win 1040, options [nop,nop,TS val 478102 ecr 2829000022], length 1448
17:02:42.331807 IP 10.0.4.2.30216 > 10.0.4.1.10449: Flags [.], ack 2897, win 1018, options [nop,nop,TS val 2829000023 ecr 478102], length 0
17:02:42.331809 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq 2897:4345, ack 1, win 1040, options [nop,nop,TS val 478102 ecr 2829000022], length 1448
17:02:42.331813 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq 4345:5793, ack 1, win 1040, options [nop,nop,TS val 478102 ecr 2829000022], length 1448
17:02:42.331817 IP 10.0.4.2.30216 > 10.0.4.1.10449: Flags [.], ack 5793, win 1018, options [nop,nop,TS val 2829000023 ecr 478102], length 0
17:02:42.331818 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq 5793:7241, ack 1, win 1040, options [nop,nop,TS val 478102 ecr 2829000022], length 1448
17:02:42.331821 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq 7241:8689, ack 1, win 1040, options [nop,nop,TS val 478102 ecr 2829000022], length 1448
17:02:42.331825 IP 10.0.4.2.30216 > 10.0.4.1.10449: Flags [.], ack 8689, win 1018, options [nop,nop,TS val 2829000023 ecr 478102], length 0
17:02:42.331826 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq 8689:10137, ack 1, win 1040, options [nop,nop,TS val 478102 ecr 2829000022], length 1448
17:02:42.331829 IP 10.0.4.1.10449 > 10.0.4.2.30216: Flags [.], seq 10137:11585, ack 1, win 1040, options [nop,nop,TS val 478102 ecr 2829000022], length 1448
...

Doing the same trace over 10G ix interfaces shows most segments in the 8-32K range, indicating that TSO/LRO are in use (and results in 9.9G throughput).

Lars
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.freebsd.org/pipermail/freebsd-net/attachments/20151210/51aa39f6/attachment.sig>