Re: IPv6 MTU discovery - packet too big

On Mon, Dec 09, 2024 at 07:47:05PM -0300, Souji Thenria wrote:
> Hey all,
> 
> On a VPS, I want to create separate jails for most services and assign
> each jail a public IPv6 address. However, I ran into an MTU issue, where
> the external interface of the host system sends multiple ICMPv6
> messages, stating that the received packets are too big to a remote
> server I tried to connect to from inside a jail. And the other server is
> ignoring these messages.
> 
> I'm running FreeBSD 14.1-RELEASE on that server and use Bastille to
> manage my jails.
> 
> The setup is as follows:
> <epair1> -- <eapir0> -- <bridge> -- <ext_inter>
> 
> ext_inter: This interface is connected to the internet and has a public
> IPv6 address. It is NOT connected to the bridge.
> 
> bridge: The bridge acts as default gateway for the jails and has a
> public IPv6 address assigned to it.
> 
> epair0: Is a member of the bridge.
> 
> epair1: This interface is passed to the jail, and a public IPv6 address is
> assigned inside the jail.
> 
> The idea is that the jails can communicate over the bridge with each
> other, and when communicating with hosts on the internet, the traffic is
> routed over the ext_inter interface.
> All interfaces have an MTU of 1500 configured.
> 
> 
> The Problem:
> When I try to connect to, e.g. a web server, the ext_inter interface
> sends a lot of ICMPv6 packets saying:
> ICMP6, packet too big, mtu 1500, length 1240
> 
> When I make the same request from the host itself, it works without any
> issues. I suspect that this is because the ext_inter interface has the
> 'JUMBO_MTU' option set, allowing packets to pass with a larger MTU.
> However, this shouldn't happen since the bridge and epair0/1 don't have
> this option.
> 
> I can also confirm that the ICMP messages pass the firewall and reach
> the remote server. However, all servers I tried seemed to ignore that
> message and resent their packets without fragmenting them to a fitting
> size.
> 
> 
> Does anyone know what the issue might be, or have they had a similar
> problem and been able to solve it?
> 

Hi !

I'm no expert on this, but I had a similar experience.

I came across something similar when I set up pppoe on my router, where
all LAN computers have an MTU of 1500 but pppoe expects messages to fit
within 1492 or less. I solved via pf.conf on the router:

match out on any from $lan:network scrub (max-mss 1440)

The OpenBSD man page has a section touching on the topic:
https://man.openbsd.org/pppoe#MTU/MSS_NEGOTIATION

I hope this might help