IPv6 MTU discovery - packet too big

From: Souji Thenria <mail_at_souji-thenria.net>
Date: Mon, 09 Dec 2024 22:47:05 UTC
Hey all,

On a VPS, I want to create separate jails for most services and assign
each jail a public IPv6 address. However, I ran into an MTU issue, where
the external interface of the host system sends multiple ICMPv6
messages, stating that the received packets are too big to a remote
server I tried to connect to from inside a jail. And the other server is
ignoring these messages.

I'm running FreeBSD 14.1-RELEASE on that server and use Bastille to
manage my jails.

The setup is as follows:
<epair1> -- <eapir0> -- <bridge> -- <ext_inter>

ext_inter: This interface is connected to the internet and has a public
IPv6 address. It is NOT connected to the bridge.

bridge: The bridge acts as default gateway for the jails and has a
public IPv6 address assigned to it.

epair0: Is a member of the bridge.

epair1: This interface is passed to the jail, and a public IPv6 address is
assigned inside the jail.

The idea is that the jails can communicate over the bridge with each
other, and when communicating with hosts on the internet, the traffic is
routed over the ext_inter interface.
All interfaces have an MTU of 1500 configured.


The Problem:
When I try to connect to, e.g. a web server, the ext_inter interface
sends a lot of ICMPv6 packets saying:
ICMP6, packet too big, mtu 1500, length 1240

When I make the same request from the host itself, it works without any
issues. I suspect that this is because the ext_inter interface has the
'JUMBO_MTU' option set, allowing packets to pass with a larger MTU.
However, this shouldn't happen since the bridge and epair0/1 don't have
this option.

I can also confirm that the ICMP messages pass the firewall and reach
the remote server. However, all servers I tried seemed to ignore that
message and resent their packets without fragmenting them to a fitting
size.


Does anyone know what the issue might be, or have they had a similar
problem and been able to solve it?

Regards,
Souji


--
Souji Thenria
Website: www.souji-thenria.net