TCP packet out-of-order problem

[Lin Jui-Nan Eric]

Dear listers,

We recently found our new FreeBSD server (located in some foreign
region) has poor network performance. After doing some tcpdump and
iperf testing, we found that out-of-order TCP packets are not inserted
into queue.

This is an 100Mbps line, and TSO is disabled.

% uname -a
FreeBSD bsd 7.1-RC2 FreeBSD 7.1-RC2 #2: Wed Dec 31 03:12:39 CST 2008
  root at bsd:/usr/obj/usr/src/sys/KERNEL  amd64

% iperf -c 10.1.1.250
------------------------------------------------------------
Client connecting to office, TCP port 5001
TCP window size: 3.07 MByte (default)
------------------------------------------------------------
[  4] local 10.1.1.210 port 61488 connected with 10.1.1.250 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.2 sec  5.74 MBytes  4.74 Mbits/sec

03:47:21.146397 IP 10.1.1.210.54919 > 10.1.1.250.5001: .
159305:160753(1448) ack 1 win 1040 <nop,nop,timestamp 555928950
3216612488>
03:47:21.146409 IP 10.1.1.250.5001 > 10.1.1.210.54919: . ack 160753
win 12568 <nop,nop,timestamp 3216612488 555928950>
03:47:21.146473 IP 10.1.1.210.54919 > 10.1.1.250.5001: .
160753:162201(1448) ack 1 win 1040 <nop,nop,timestamp 555928950
3216612488>
03:47:21.146485 IP 10.1.1.250.5001 > 10.1.1.210.54919: . ack 162201
win 12568 <nop,nop,timestamp 3216612489 555928950>
03:47:21.146972 IP 10.1.1.210.54919 > 10.1.1.250.5001: .
163649:165097(1448) ack 1 win 1040 <nop,nop,timestamp 555928950
3216612488>
03:47:21.146983 IP 10.1.1.250.5001 > 10.1.1.210.54919: . ack 162201
win 12573 <nop,nop,timestamp 3216612489 555928950>
03:47:21.146985 IP 10.1.1.210.54919 > 10.1.1.250.5001: .
162201:163649(1448) ack 1 win 1040 <nop,nop,timestamp 555928950
3216612488>
03:47:21.146996 IP 10.1.1.250.5001 > 10.1.1.210.54919: . ack 163649
win 12568 <nop,nop,timestamp 3216612489 555928950>
03:47:21.146998 IP 10.1.1.210.54919 > 10.1.1.250.5001: .
165097:166545(1448) ack 1 win 1040 <nop,nop,timestamp 555928950
3216612488>
03:47:21.147006 IP 10.1.1.250.5001 > 10.1.1.210.54919: . ack 163649
win 12573 <nop,nop,timestamp 3216612489 555928950>
03:47:21.147009 IP 10.1.1.210.54919 > 10.1.1.250.5001: .
166545:167993(1448) ack 1 win 1040 <nop,nop,timestamp 555928950
3216612488>
03:47:21.147017 IP 10.1.1.250.5001 > 10.1.1.210.54919: . ack 163649
win 12573 <nop,nop,timestamp 3216612489 555928950>
03:47:21.147019 IP 10.1.1.210.54919 > 10.1.1.250.5001: .
167993:169441(1448) ack 1 win 1040 <nop,nop,timestamp 555928950
3216612488>

* You can see "ack 163649" repeating, but the packet is transmitted
before 163649:165097.

% cat /etc/sysctl.conf
# $FreeBSD: src/etc/sysctl.conf,v 1.8 2003/03/13 18:43:50 mux Exp $
#
#  This file is read when going to multi-user and its contents piped thru
#  ``sysctl'' to adjust kernel values.  ``man 5 sysctl.conf'' for details.
#

# Uncomment this to prevent users from seeing information about processes that
# are being run under another UID.
#security.bsd.see_other_uids=0
debug.bootverbose=1
kern.ipc.somaxconn=8192
kern.maxfiles=65536
kern.maxfilesperproc=32768
kern.maxprocperuid=65536
net.inet.ip.fastforwarding=1
net.inet.tcp.delayed_ack=0
vm.pmap.shpgperproc=2000
kern.ipc.maxsockbuf=8388608
net.inet.tcp.sendspace=3217968
net.inet.tcp.recvspace=3217968

Is our configuration wrong? Or it is an known bug? I have searched
stable & net list, but found no similar discussion.
Thank you all in advance!