TSO help or hindrance ? (was Re: TSO and FreeBSD vs Linux)
Mike Tancsa
mike at sentex.net
Wed Sep 4 15:29:08 UTC 2013
On 9/4/2013 8:50 AM, Rick Macklem wrote:
> David Wolfskill wrote:
>>
>>
>> I noticed that when I tried to write files to NFS, I could write
>> small
>> files OK, but larger ones seemed to ... hang.
>> * "ifconfig -v em0" showed flags TSO4 & VLAN_HWTSO turned on.
>> * "sysctl net.inet.tcp.tso" showed "1" -- enabled.
>>
>> As soon as I issued "sudo net.inet.tcp.tso=0" ... the copy worked
>> without
>> a hitch or a whine. And I was able to copy all 117709618 bytes, not
>> just
>> 2097152 (2^21).
>>
>> Is the above expected? It came rather as a surprise to me.
>>
> Not surprising to me, I'm afraid. When there are serious NFS problems
> like this, it is often caused by a network fabric issue and broken
> TSO is at the top of the list w.r.t. cause.
I was just experimenting a bit with iSCSI via FreeNAS and was a little
disappointed at the speeds I was getting. So, I tried disabling tso on
both boxes and it did seem to speed things up a bit. Data and testing
methods attached in a txt file.
I did 3 cases.
Just boot up FreeNAS and the initiator without tweaks. That had the
worst performance.
disable tso on the nic as well as via sysctl on both boxes. That had the
best performance.
re-enable tso on both boxes. That had better performance than the first
case, but still not as good as totally disabling it. I am guessing
something is not quite being re-enabled properly ? But its different
than the other two cases ?!?
tgt is FreeNAS-9.1.1-RELEASE-x64 (a752d35) and initiator is r254328 9.2
AMD64
The FreeNAS box has 16G of RAM, so the file is being served out of cache
as gstat shows no activity when sending out the file
---Mike
--
-------------------
Mike Tancsa, tel +1 519 651 3400
Sentex Communications, mike at sentex.net
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada http://www.tancsa.com/
-------------- next part --------------
3 data files.
notso = tso disabled on tgt and initiator
tso-boot = initiator is rebooted and tests are run
tso = tso disabled and then re-enabled. For whatever reason its not as bad as post boot, but still worse than no tso
0{mdttestbox}# ministat notso tso-boot
x notso
+ tso-boot
+----------------------------------------------------------------------------------------------------------------------+
| + |
| +++ x |
|+ ++++ xxxx x x x x|
| |AM| |___M__A_____| |
+----------------------------------------------------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 9 69086843 71749085 69475320 69810666 887628.9
+ 9 55846191 56473897 56357956 56302165 208843.3
Difference at 95.0% confidence
-1.35085e+07 +/- 644386
-19.3502% +/- 0.923048%
(Student's t, pooled s = 644787)
0{mdttestbox}#
# ministat tso notso
x tso
+ notso
+----------------------------------------------------------------------------------------------------------------------+
| x + |
|x xxxxx x + x + + + + + + +|
| |____MA___||___________________M___________A_________________________________| |
+----------------------------------------------------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 9 68632067 69126677 68779201 68808473 142050.01
+ 9 69086843 71749085 69475320 69810666 887628.9
Difference at 95.0% confidence
1.00219e+06 +/- 635239
1.4565% +/- 0.923199%
(Student's t, pooled s = 635635)
0{mdttestbox}# cat tso
68779201
68734143
68915705
68752827
68782212
69126677
68828520
68724901
68632067
0{mdttestbox}# cat notso
71749085
69256608
69097532
69086843
70587459
69179672
69475320
69754511
70108963
0{mdttestbox}# cat tso-boot
55846191
56314173
56284204
56095729
56466769
56446535
56357956
56434027
56473897
0{mdttestbox}#
0{mdttestbox}# cat t.sh
#!/bin/sh
command="dd if=/mnt/test of=/dev/null bs=4096k"
for i in `jot 9 1`;do
mount /dev/da0a /mnt
eval $command 2>&1 | grep bytes | awk -F"[\( ]" '{print $8}'
umount /mnt
done
0{mdttestbox}#
initiator is r254328 9.2 AMD64
The two boxes are linked via private gigE switch
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=4209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWTSO>
ether 00:1e:67:49:31:b8
inet 10.254.1.2 netmask 0xffffff00 broadcast 10.254.1.255
inet6 fe80::21e:67ff:fe49:31b8%em1 prefixlen 64 scopeid 0x3
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
ifconfig igb0
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
ether 00:15:17:fe:a5:bc
inet 10.254.1.1 netmask 0xffffff00 broadcast 10.254.1.255
inet6 fe80::215:17ff:fefe:a5bc%igb0 prefixlen 64 scopeid 0x1
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
Initiator just does iscontrol -c /etc/iscsi.conf -n freebsd
freebsd {
authmethod = none
initiatorname = mdttestbox.sentex.ca
TargetName = iqn.2011-03.org.example.istgt:freebsd
TargetAddress = 10.254.1.1:3260,2
}
ifconfig igb0 -tso
ifconfig em0 -tso
and
sysctl -w net.inet.tcp.tso=0
More information about the freebsd-net
mailing list