[IPsec] Weird performance issue via IPsec/racoon tunnel

[Michael Grimm]

Hi

I do run an IPsec/racoon tunnel between two servers (11.1-STABLE #0 r326663). Some days ago I did migrate one of my servers from bare metal to a public cloud instance. Now I do observe weird performance issues from new to old server:

ifconfig (OLD server, bare metal):
	ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,\
			TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>

ifconfig (NEW server, cloud instance):
	vtnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,\
			TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>

Immediately after booting of NEW (test file has 10 MB) I do observe the following:

	#) scp OLD to NEW via ssh/internet:	16.7 MB/s
	#) scp NEW to OLD via ssh/internet:	17.4 MB/s
	#) scp NEW to OLD via IPsec tunnel:	-> 65.8 KB/s !
	#) scp OLD to NEW via IPsec tunnel:	16.5 MB/s

Now I do a "ifconfig vtnet0 mtu 1500 up" and can observe very similar performance.

*BUT* if I do a "ifconfig vtnet0 mtu 1450 up ; ifconfig vtnet0 mtu 1500 up" I do observe:

	#) scp NEW to OLD via IPsec tunnel:	17.1 MB/s !
	#) scp OLD to NEW via IPsec tunnel:	16.9 MB/s

I did monitor "tcpdump -i ix0 -vv esp" at the OLD sever and do get many:

	16:22:24.370486 IP (tos 0x8, ttl 64, id 17394, offset 0, flags [none], proto ESP (50), \
			length 140, bad cksum 0 (->b110)!)
	    "OLD" > "NEW": ESP(spi=0x0d83dae4,seq=0x3a8d9a), length 120

At the NEW server I do not observe those checksum errors at all. *BUT* I do see these error even after regaining full performance by modifying the MTU from 1500 to 1450 and back to 1500!

Well, I do have to admit that I do not have enough knowledge about networking to find out by myself what to debug/modify next.

Any help is highly appreciated.

Thanks in advance,
Michael