Path MTU discovery broken in IPSec

[Khetan Gajjar]

Hi.

Summary; searching for this problem revealed another query, but no
solution -
http://lists.freebsd.org/pipermail/freebsd-net/2005-July/007899.html

Explanation;
I'm experiencing a broken path MTU discovery problem between two
hosts connecting with each other via IPSec transport mode, exasperated
by the fact that the two hosts are more than 600ms apart in terms
of network latency.

Host 1 and Host 2 both run FreeBSD 6.1-stable, circa Sep 7.

Host 1's IPsec config looks like
/etc/ipsec.conf:
flush;
spdflush;
spdadd x.x.x.x y.y.y.y any -P out ipsec esp/transport//require;
spdadd y.y.y.y x.x.x.x any -P in ipsec esp/transport//require;

and its network config looks like
em0: flags=9843<UP,BROADCAST,RUNNING,SIMPLEX,LINK0,MULTICAST> mtu 1500
         options=b<RXCSUM,TXCSUM,VLAN_MTU>
         inet6 fe80::212:3fff:feec:d1ce%em0 prefixlen 64 scopeid 0x1
         inet x.x.x.x netmask 0xffffff00 broadcast x.x.x.255
         ether 00:12:3f:ec:d1:ce
         media: Ethernet 100baseTX <full-duplex>
         status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
         inet6 ::1 prefixlen 128
         inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
         inet 127.0.0.1 netmask 0xff000000

Host 2's IPsec config looks like
/etc/ipsec.conf:
flush;
spdflush;
spdadd x.x.x.x y.y.y.y any -P in ipsec esp/transport//require;
spdadd y.y.y.y x.x.x.x any -P out ipsec esp/transport//require;

and its network config looks like
fxp0: flags=9843<UP,BROADCAST,RUNNING,SIMPLEX,LINK0,MULTICAST> mtu 1500
         options=b<RXCSUM,TXCSUM,VLAN_MTU>
         inet6 fe80::202:b3ff:feeb:21db%fxp0 prefixlen 64 scopeid 0x1
         inet y.y.y.y netmask 0xfffffff8 broadcast y.y.y.z
         ether 00:02:b3:eb:21:db
         media: Ethernet 10baseT/UTP <full-duplex>
         status: active
plip0: flags=108810<POINTOPOINT,SIMPLEX,MULTICAST,NEEDSGIANT> mtu 1500
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
         inet6 ::1 prefixlen 128
         inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
         inet 127.0.0.1 netmask 0xff000000

Both machines are running the same kernel configs and the same
sysctl configs. The sysctl's in play are
net.inet.icmp.icmplim=500
net.inet.ip.ttl=128
net.inet.raw.maxdgram=57344
net.inet.raw.recvspace=65535
net.inet.tcp.always_keepalive=1
net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1
net.inet.icmp.drop_redirect=1
net.inet.icmp.log_redirect=1
net.inet.ip.redirect=0
net.inet6.ip6.redirect=0
net.inet.ip.sourceroute=0
net.inet.ip.accept_sourceroute=0
net.inet.icmp.bmcastecho=0
net.inet.icmp.maskrepl=0
net.inet.tcp.delayed_ack=0
net.inet.tcp.sendspace=65535
net.inet.tcp.recvspace=65535
net.inet.udp.recvspace=65535
net.inet.udp.maxdgram=57344
net.local.stream.recvspace=65535
net.local.stream.sendspace=65535

racoon does its thing, and the ipsec tunnels come up. I can ping
both sides, and there are no ipfw rules running. Connectivity via
ssh and nfs seems to work fine, as do DNS zone transfers (for very
small zones).

Connectivity from host 2 to host 1 works perfectly. From host 1
to host 2 however, TCP sessions break / stall / timeout. I've tried
reducing the MTU sizes from the default 1500 to 1492 on both
interfaces, and that makes no difference.

Are there any suggestions or additional debugging that could assist
in solving this problem ?

Khetan Gajjar.
--
khetan at os.org.za
+27 82 885 4047