Some hosting weirdness...

Wed Jul 11 18:38:45 UTC 2007

On Wednesday 11 July 2007 20:14:59 Eric F Crist wrote:
> Well, I performed a tcpdump as you suggested, and my mss is exactly
> 1460, not the 1452 you suggest.  What does this mean?

As your servers uplink is (most probably) an Ethernet cable, your MSS is 
correct at 1460 (= 1500 bytes MTU for Ethernet - 40 bytes IP+TCP header).

When a TCP connection is established, a three-way handshake takes place. The 
host opening the connection sends a SYN-packet which contains "his" Maximum 
Segment Size, in this case it's the customer opening a website on your 
server, and your host sends a confirmation SYN/ACK-packet to open your side 
of the two way connection, which contains your MSS. This makes two values for 
Maximum Segment Size (the remote one and yours), and the smaller one is 
chosen as the Maximum Segment Size of the connection, thus if the customer 
sends a SYN-packet with MSS of 1452 and you send back a SYN/ACK with MSS of 
1460, the MSS for the connection is negotiated at 1452 (which both hosts 
should stick to).

The following TCP dump of a connection request to a host (sadly a Linux 
box ;-)) should clear any confusion:

root at beenic01:/home/heiko# tcpdump -vv -i eth0 port 80 and host 
hnvr-4db2ebb3.pool.einsundeins.de
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes

--- SYN packet from my dialup (MSS of 1452, I'm on DSL)
20:22:26.329522 IP (tos 0x0, ttl  52, id 8003, offset 0, flags [DF], proto: 
TCP (6), length: 64) hnvr-4db2ebb3.pool.einsundeins.de.64905 > 
mail.beenic.net.www: S, cksum 0xd2b5 (correct), 1315765383:1315765383(0) win 
65535 <mss 1452,nop,wscale 0,nop,nop,timestamp 2442717 0,sackOK,eol>
---

--- SYN/ACK from server (MSS of 1460, is on 100Mbit Ethernet)
20:22:26.331590 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], proto: TCP 
(6), length: 44) mail.beenic.net.www > 
hnvr-4db2ebb3.pool.einsundeins.de.64905: S, cksum 0x421a (correct), 
1939516734:1939516734(0) ack 1315765384 win 5840 <mss 1460>
---

--- Some connection setup
20:22:26.395813 IP (tos 0x0, ttl  52, id 8004, offset 0, flags [DF], proto: 
TCP (6), length: 40) hnvr-4db2ebb3.pool.einsundeins.de.64905 > 
mail.beenic.net.www: ., cksum 0x70a7 (correct), 1:1(0) ack 1 win 65535
20:22:26.402403 IP (tos 0x0, ttl  52, id 8005, offset 0, flags [DF], proto: 
TCP (6), length: 421) hnvr-4db2ebb3.pool.einsundeins.de.64905 > 
mail.beenic.net.www: P 1:382(381) ack 1 win 65535
20:22:26.402414 IP (tos 0x0, ttl  64, id 58600, offset 0, flags [DF], proto: 
TCP (6), length: 40) mail.beenic.net.www > 
hnvr-4db2ebb3.pool.einsundeins.de.64905: ., cksum 0x560a (correct), 1:1(0) 
ack 382 win 6432
---

--- Actual data packet (IP packet size is the smaller of the two MSS+40)
20:22:26.923728 IP (tos 0x0, ttl  64, id 58602, offset 0, flags [DF], proto: 
TCP (6), length: 1492) mail.beenic.net.www > 
hnvr-4db2ebb3.pool.einsundeins.de.64905: . 1:1453(1452) ack 382 win 6432
---

--- Another data packet (again, smaller of the two MSS+40 bytes)
20:22:26.923739 IP (tos 0x0, ttl  64, id 58604, offset 0, flags [DF], proto: 
TCP (6), length: 1492) mail.beenic.net.www > 
hnvr-4db2ebb3.pool.einsundeins.de.64905: . 1453:2905(1452) ack 382 win 6432
---

And so on and so forth... This output was grabbed while I was loading an HTML 
page from the server which is around 5kb large, which means that at least one 
TCP packet is filled up completely.

Ping also makes it easy to spot this:

root at beenic01:/home/heiko# ping -s 1464 hnvr-4db2ebb3.pool.einsundeins.de
PING hnvr-4db2ebb3.pool.einsundeins.de (77.178.235.179) 1464(1492) bytes of 
data.
1472 bytes from hnvr-4db2ebb3.pool.einsundeins.de (77.178.235.179): icmp_seq=1 
ttl=53 time=193 ms
1472 bytes from hnvr-4db2ebb3.pool.einsundeins.de (77.178.235.179): icmp_seq=2 
ttl=53 time=191 ms
1472 bytes from hnvr-4db2ebb3.pool.einsundeins.de (77.178.235.179): icmp_seq=3 
ttl=53 time=188 ms
1472 bytes from hnvr-4db2ebb3.pool.einsundeins.de (77.178.235.179): icmp_seq=4 
ttl=53 time=191 ms

--- hnvr-4db2ebb3.pool.einsundeins.de ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3002ms
rtt min/avg/max/mdev = 188.379/191.356/193.704/1.912 ms
root at beenic01:/home/heiko#

1464 ping bytes (making a total IP+ICMP packet size of 1492) fit through the 
pipe, but:

root at beenic01:/home/heiko# ping -s 1465 hnvr-4db2ebb3.pool.einsundeins.de
PING hnvr-4db2ebb3.pool.einsundeins.de (77.178.235.179) 1465(1493) bytes of 
data.
From rtsl-hnvr-de05.nw.mediaways.net (213.20.127.85) icmp_seq=1 Frag needed 
and DF set (mtu = 1492)
1473 bytes from hnvr-4db2ebb3.pool.einsundeins.de (77.178.235.179): icmp_seq=2 
ttl=53 time=180 ms
1473 bytes from hnvr-4db2ebb3.pool.einsundeins.de (77.178.235.179): icmp_seq=3 
ttl=53 time=179 ms
1473 bytes from hnvr-4db2ebb3.pool.einsundeins.de (77.178.235.179): icmp_seq=4 
ttl=53 time=202 ms
1473 bytes from hnvr-4db2ebb3.pool.einsundeins.de (77.178.235.179): icmp_seq=5 
ttl=53 time=198 ms
1473 bytes from hnvr-4db2ebb3.pool.einsundeins.de (77.178.235.179): icmp_seq=6 
ttl=53 time=174 ms

--- hnvr-4db2ebb3.pool.einsundeins.de ping statistics ---
17 packets transmitted, 5 received, +1 errors, 70% packet loss, time 16003ms
rtt min/avg/max/mdev = 174.848/186.982/202.520/11.114 ms
root at beenic01:/home/heiko#

1465 ping bytes don't, or only do if fragmented: see the ping command output 
first message, where an infrastructure router of my ISP which takes care of 
routing the packet to me tells the pinging host (the server) that 1493 
IP-bytes won't fit through the pipe to me, at least not if the packet is not 
to be fragmented (which the DF flag in the IP header signifies). ping changes 
the IP header to allow fragmentation from ICMP ping packet 2 on, and the 
router happily starts fragmenting the packets for me, to which my host then 
starts replying.

Finally my ISP stops fragmenting the packet (probably because of some routing 
switch) after packet 6, and the packets don't come through to me anymore, 
because the new router also doesn't fragment (and doesn't send an ICMP error 
either, like the router I got for the first 6 packets did).

The ping test is ideal for testing the hypothesis of an MSS problem, which can 
easily arise if some router/firewall between your host and the destination 
does packet mangling on the MSS on connection setup, which has bitten me more 
than once, especially in the presence of VLAN technology, which shrinks the 
MTU of the Ethernet interface to 1496 bytes, making an MSS of 1456.

Anyway, hope this helps for now.

-- 
Heiko Wundram
Product & Application Development
-------------------------------------
Office Germany - EXPO PARK HANNOVER

Beenic Networks GmbH
Mailänder Straße 2
30539 Hannover

Fon        +49 511 / 590 935 - 15
Fax        +49 511 / 590 935 - 29
Mail       wundram at beenic.net

Beenic Networks GmbH
-------------------------------------
Sitz der Gesellschaft: Hannover
Geschäftsführer: Jorge Delgado
Registernummer: HRB 61869
Registergericht: Amtsgericht Hannover