Terrible NFS performance under 9.2-RELEASE?

Fri Jan 24 22:59:12 UTC 2014

J David wrote:
> Here's a pair of quick tcpdumps with TSO/LRO on and off showing that
> tcpdump does not reassemble small packets into larger ones:
> 
> TSO/LRO on:
> 
> 05:21:54.956061 IP 172.20.20.169.24265 > 172.20.20.162.5001: Flags
> [S], seq 2538331145, win 65535, options [mss 1460,nop,wscale
> 9,sackOK,TS val 2932122 ecr 0], length 0
> 05:21:54.956239 IP 172.20.20.162.5001 > 172.20.20.169.24265: Flags
> [S.], seq 3371756775, ack 2538331146, win 65535, options [mss
> 1460,nop,wscale 9,sackOK,TS val 2833562423 ecr 2932122], length 0
> 05:21:54.956292 IP 172.20.20.169.24265 > 172.20.20.162.5001: Flags
> [.], ack 1, win 2050, options [nop,nop,TS val 2932132 ecr
> 2833562423],
> length 0
> 05:21:54.956372 IP 172.20.20.169.24265 > 172.20.20.162.5001: Flags
> [P.], seq 1:25, ack 1, win 2050, options [nop,nop,TS val 2932132 ecr
> 2833562423], length 24
> 05:21:54.956432 IP 172.20.20.162.5001 > 172.20.20.169.24265: Flags
> [.], ack 1, win 2050, options [nop,nop,TS val 2833562423 ecr
> 2932132],
> length 0
> 05:21:54.956495 IP 172.20.20.169.24265 > 172.20.20.162.5001: Flags
> [.], seq 25:4381, ack 1, win 2050, options [nop,nop,TS val 2932132
> ecr
> 2833562423], length 4356
> 05:21:54.956604 IP 172.20.20.162.5001 > 172.20.20.169.24265: Flags
> [.], ack 4381, win 2041, options [nop,nop,TS val 2833562423 ecr
> 2932132], length 0
> 05:21:54.956620 IP 172.20.20.169.24265 > 172.20.20.162.5001: Flags
> [.], seq 4381:11657, ack 1, win 2050, options [nop,nop,TS val 2932132
> ecr 2833562423], length 7276
> 05:21:55.050656 IP 172.20.20.162.5001 > 172.20.20.169.24265: Flags
> [.], ack 11657, win 2050, options [nop,nop,TS val 2833562523 ecr
> 2932132], length 0
> 05:21:55.050686 IP 172.20.20.169.24265 > 172.20.20.162.5001: Flags
> [.], seq 11657:21829, ack 1, win 2050, options [nop,nop,TS val
> 2932222
> ecr 2833562523], length 10172
> 05:21:55.150644 IP 172.20.20.162.5001 > 172.20.20.169.24265: Flags
> [.], ack 21829, win 2050, options [nop,nop,TS val 2833562623 ecr
> 2932222], length 0
> 05:21:55.150674 IP 172.20.20.169.24265 > 172.20.20.162.5001: Flags
> [.], seq 21829:34897, ack 1, win 2050, options [nop,nop,TS val
> 2932322
> ecr 2833562623], length 13068
> 05:21:55.250629 IP 172.20.20.162.5001 > 172.20.20.169.24265: Flags
> [.], ack 34897, win 2050, options [nop,nop,TS val 2833562723 ecr
> 2932322], length 0
> 05:21:55.250658 IP 172.20.20.169.24265 > 172.20.20.162.5001: Flags
> [.], seq 34897:50861, ack 1, win 2050, options [nop,nop,TS val
> 2932422
> ecr 2833562723], length 15964
> 05:21:55.350647 IP 172.20.20.162.5001 > 172.20.20.169.24265: Flags
> [.], ack 50861, win 2050, options [nop,nop,TS val 2833562823 ecr
> 2932422], length 0
> 05:21:55.350677 IP 172.20.20.169.24265 > 172.20.20.162.5001: Flags
> [.], seq 50861:69721, ack 1, win 2050, options [nop,nop,TS val
> 2932522
> ecr 2833562823], length 18860
> 05:21:55.450656 IP 172.20.20.162.5001 > 172.20.20.169.24265: Flags
> [.], ack 69721, win 2050, options [nop,nop,TS val 2833562923 ecr
> 2932522], length 0
> 05:21:55.450686 IP 172.20.20.169.24265 > 172.20.20.162.5001: Flags
> [.], seq 69721:91477, ack 1, win 2050, options [nop,nop,TS val
> 2932622
> ecr 2833562923], length 21756
> 05:21:55.550577 IP 172.20.20.162.5001 > 172.20.20.169.24265: Flags
> [.], ack 91477, win 2050, options [nop,nop,TS val 2833563023 ecr
> 2932622], length 0
> 05:21:55.550608 IP 172.20.20.169.24265 > 172.20.20.162.5001: Flags
> [.], seq 91477:116129, ack 1, win 2050, options [nop,nop,TS val
> 2932722 ecr 2833563023], length 24652
> 05:21:55.650645 IP 172.20.20.162.5001 > 172.20.20.169.24265: Flags
> [.], ack 116129, win 2050, options [nop,nop,TS val 2833563123 ecr
> 2932722], length 0
> 05:21:55.650676 IP 172.20.20.169.24265 > 172.20.20.162.5001: Flags
> [.], seq 116129:143677, ack 1, win 2050, options [nop,nop,TS val
> 2932822 ecr 2833563123], length 27548
> 05:21:55.750643 IP 172.20.20.162.5001 > 172.20.20.169.24265: Flags
> [.], ack 143677, win 2050, options [nop,nop,TS val 2833563223 ecr
> 2932822], length 0
> 05:21:55.750675 IP 172.20.20.169.24265 > 172.20.20.162.5001: Flags
> [.], seq 143677:174121, ack 1, win 2050, options [nop,nop,TS val
> 2932922 ecr 2833563223], length 30444
> 05:21:55.850636 IP 172.20.20.162.5001 > 172.20.20.169.24265: Flags
> [.], ack 174121, win 2050, options [nop,nop,TS val 2833563323 ecr
> 2932922], length 0
> 05:21:55.850667 IP 172.20.20.169.24265 > 172.20.20.162.5001: Flags
> [.], seq 174121:207461, ack 1, win 2050, options [nop,nop,TS val
> 2933022 ecr 2833563323], length 33340
> 
> TSO/LRO off:
> 
> 05:19:34.556302 IP 172.20.20.169.53485 > 172.20.20.162.5001: Flags
> [S], seq 529322163, win 65535, options [mss 1460,nop,wscale
> 9,sackOK,TS val 2791722 ecr 0], length 0
> 05:19:34.556414 IP 172.20.20.162.5001 > 172.20.20.169.53485: Flags
> [S.], seq 3835815533, ack 529322164, win 65535, options [mss
> 1460,nop,wscale 9,sackOK,TS val 1931664416 ecr 2791722], length 0
> 05:19:34.556443 IP 172.20.20.169.53485 > 172.20.20.162.5001: Flags
> [.], ack 1, win 2050, options [nop,nop,TS val 2791732 ecr
> 1931664416],
> length 0
> 05:19:34.556505 IP 172.20.20.169.53485 > 172.20.20.162.5001: Flags
> [P.], seq 1:25, ack 1, win 2050, options [nop,nop,TS val 2791732 ecr
> 1931664416], length 24
> 05:19:34.556604 IP 172.20.20.169.53485 > 172.20.20.162.5001: Flags
> [.], seq 25:1473, ack 1, win 2050, options [nop,nop,TS val 2791732
> ecr
> 1931664416], length 1448
> 05:19:34.556621 IP 172.20.20.162.5001 > 172.20.20.169.53485: Flags
> [.], ack 1, win 2050, options [nop,nop,TS val 1931664416 ecr
> 2791732],
> length 0
> 05:19:34.556648 IP 172.20.20.169.53485 > 172.20.20.162.5001: Flags
> [.], seq 1473:2921, ack 1, win 2050, options [nop,nop,TS val 2791732
> ecr 1931664416], length 1448
> 05:19:34.556672 IP 172.20.20.169.53485 > 172.20.20.162.5001: Flags
> [.], seq 2921:4369, ack 1, win 2050, options [nop,nop,TS val 2791732
> ecr 1931664416], length 1448
> 05:19:34.556711 IP 172.20.20.162.5001 > 172.20.20.169.53485: Flags
> [.], ack 1473, win 2047, options [nop,nop,TS val 1931664416 ecr
> 2791732], length 0
> 05:19:34.556730 IP 172.20.20.169.53485 > 172.20.20.162.5001: Flags
> [.], seq 4369:5817, ack 1, win 2050, options [nop,nop,TS val 2791732
> ecr 1931664416], length 1448
> 05:19:34.556743 IP 172.20.20.169.53485 > 172.20.20.162.5001: Flags
> [.], seq 5817:7265, ack 1, win 2050, options [nop,nop,TS val 2791732
> ecr 1931664416], length 1448
> 05:19:34.556755 IP 172.20.20.162.5001 > 172.20.20.169.53485: Flags
> [.], ack 4369, win 2041, options [nop,nop,TS val 1931664416 ecr
> 2791732], length 0
> 05:19:34.556774 IP 172.20.20.169.53485 > 172.20.20.162.5001: Flags
> [.], seq 7265:8713, ack 1, win 2050, options [nop,nop,TS val 2791732
> ecr 1931664416], length 1448
> 05:19:34.556793 IP 172.20.20.169.53485 > 172.20.20.162.5001: Flags
> [.], seq 8713:10161, ack 1, win 2050, options [nop,nop,TS val 2791732
> ecr 1931664416], length 1448
> 05:19:34.556813 IP 172.20.20.169.53485 > 172.20.20.162.5001: Flags
> [.], seq 10161:11609, ack 1, win 2050, options [nop,nop,TS val
> 2791732
> ecr 1931664416], length 1448
> 05:19:34.556830 IP 172.20.20.169.53485 > 172.20.20.162.5001: Flags
> [.], seq 11609:13057, ack 1, win 2050, options [nop,nop,TS val
> 2791732
> ecr 1931664416], length 1448
> 05:19:34.556848 IP 172.20.20.162.5001 > 172.20.20.169.53485: Flags
> [.], ack 7265, win 2036, options [nop,nop,TS val 1931664416 ecr
> 2791732], length 0
> 05:19:34.556865 IP 172.20.20.169.53485 > 172.20.20.162.5001: Flags
> [.], seq 13057:14505, ack 1, win 2050, options [nop,nop,TS val
> 2791732
> ecr 1931664416], length 1448
> 05:19:34.556881 IP 172.20.20.169.53485 > 172.20.20.162.5001: Flags
> [.], seq 14505:15953, ack 1, win 2050, options [nop,nop,TS val
> 2791732
> ecr 1931664416], length 1448
> 05:19:34.556893 IP 172.20.20.169.53485 > 172.20.20.162.5001: Flags
> [.], seq 15953:17401, ack 1, win 2050, options [nop,nop,TS val
> 2791732
> ecr 1931664416], length 1448
> 05:19:34.556912 IP 172.20.20.169.53485 > 172.20.20.162.5001: Flags
> [.], seq 17401:18849, ack 1, win 2050, options [nop,nop,TS val
> 2791732
> ecr 1931664416], length 1448
> 05:19:34.556929 IP 172.20.20.162.5001 > 172.20.20.169.53485: Flags
> [.], ack 10161, win 2030, options [nop,nop,TS val 1931664416 ecr
> 2791732], length 0
> 
> The stated lengths represent the length of each IP packet as it
> appears on the "wire."  The result is the same whether it is obtained
> from the client, the server, or the KVM host snooping directly on the
> bridge.
> 
Yes, although no real wire would ever have packets this big. I now realize
that you have a virtual pseudo ethernet that handles 64K packets. Sorry
for the confusion this has caused.

Since I know absolutely nothing about what the effects of such a virtual
device are, I will be useless to you in helping with this. (I don't even
know if wireshark could figure this out?)

rick

> Thanks!
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to
> "freebsd-net-unsubscribe at freebsd.org"
> 