RE: PPPoE (mpd5) IPv6 issues

> -----Original Message-----
> From: driesm@FreeBSD.org <driesm@FreeBSD.org>
> Sent: Saturday, 17 September 2022 10:18
> To: 'Evilham' <contact@evilham.com>
> Cc: freebsd-net@freebsd.org
> Subject: RE: PPPoE (mpd5) IPv6 issues
> 
> > -----Original Message-----
> > From: Evilham <contact@evilham.com>
> > Sent: Friday, 16 September 2022 22:00
> > To: driesm@FreeBSD.org
> > Cc: freebsd-net@freebsd.org
> > Subject: Re: PPPoE (mpd5) IPv6 issues
> >
> > Hey,
> >
> > On dv., set. 16 2022, driesm@FreeBSD.org wrote:
> >
> > > Hmm this mail was not finished, sorry about that.
> > >
> > > I will include the link that fell-off, IPv6 PPPoE MSS incorrect
> > > | Netgate Forum
> > >
> > >
> > >
> > > Any help or pointers are greatly appreciated!
> > >
> > >
> > >
> > > PS: I use IPFW as my firewall, if this has anything to do with it. I
> > > do have a “reassemble all in” rule at the top of my ruleset
> > >
> > >
> > >
> > > From: driesm@FreeBSD.org <driesm@FreeBSD.org>
> > > Sent: Friday, 16 September 2022 18:36
> > > To: freebsd-net@freebsd.org
> > > Subject: PPPoE (mpd5) IPv6 issues
> > >
> > >
> > >
> > > Hi freebsd-net!
> > >
> > >
> > >
> > > After a lot of google and tinkering I have hit a brick wall trying
> > > to set-up my new ISP which uses PPPoE.
> > >
> > > I’m at the point where IPv4 works fine 100% for all websites.
> > > But from the moment I turn on IPv6 all sites that prefer IPv6 stop
> > > working, I have confirmed that routing table etc all look good. So
> > > I’m more suspicious to a MTU / MSS issue.
> > >
> > > Although I cant find that much info about it. I have tcpmss fix
> > > turned on in my mpd.conf
> > >
> > >
> > >
> > > There seems to be a few sites like google.com and youtube.com that
> > > keep working.
> > >
> > >
> > >
> > > I have googled thoroughly but I was unable to find a fix for my
> > > problem. The one close to my issue seems  this issue from 2020:
> >
> >
> > Some things to take into account are: MTU of the physical interface
> > and ICMP blackholes, see below.
> >
> > For the MTU of the physical interface, take into account that if a
> > VLAN and PCP (vlan(4)+ifconfig(8)) are required by the wholesale
> > provider, those consume 4 bytes each and taking PPPoE's 8 bytes into
> > account as well, your physical interface should have a higher MTU (4
> > bytes for VLAN + 4 bytes for PCP + 8 bytes for PPPoE = 16 bytes
> > overhead on physical interface) than the connection (I'd guess one of
> > 1500, 1492 or 1484 depending on the wholesale provider and the ISP).
> >
> >
> > Basically, from the router if the interface created by mpd5 has an MTU
> > of 1500, you should be able to:
> >
> > ping -6D -s 1452 ipv6.icmp.host
> >
> > With -6D being "do not fragment"
> > And 1452 bytes being the payload =
> >   1500 bytes - 40 bytes from IPv6 headers - 8 bytes for ICMP
> >   header
> >
> >
> > And you **should** receive a "ping: sendmsg: Message too long"
> > with:
> >
> > ping -6D -s 1453 ipv6.icmp.host
> >
> >
> > When in doubt, try:
> > ping -6D -s 1232 ipv6.icmp.host
> > This should work since 1280 bytes is the minimum MTU for IPv6 and
> > gradually go up to establish what the maximum MTU for your connection
> > is being.
> 
> Thanks a lot for the troubleshooting steps!
> 
> After trying this I identified the max ping size to be 1444 bytes.
> 1445 bytes results in message too long. This was both the case on the router
> as one of the clients (windows), so consistency seems good here.
> 
> This was with ng0 configured on the standard 1492 bytes MTU.
> All my other physical interfaces are on the standard 1500 MTU.
> 
> Now how do I translate this info to getting IPv6 to work reliably?
> Which MTU should I tweak?  LAN side or WAN side?
> 
> Dries

Okay so after some more tinkering I did found a solution.
I disabled tcpmssfix in mpd.conf, and moved it to IPFW with one rule:
	tcp-setmss 1432 tcp from any to any out

Seems like 1492 - 40 -20 works for both IPv4 and IPv6 stacks.
I'm not sure which value is being used by mpd5 when setting up ng_tcpmss.

> 
> >
> >
> > If you don't receive the "Message too long" with way too large
> > packets, there might be a bit of an ICMP blackhole on the ISP side or
> > on the route to the ipv6.icmp.host.
> >
> > This is usually not too tragic for IPv4 because routers do the
> > fragmenting magic, but it is terrible for IPv6 since it is the client
> > that has to fragment, and for that it needs to receive the "Message
> > too long" ICMP response.
> >
> > By using tcpdump on the router, convince yourself truly that it is not
> > *you* dropping all incoming ICMP! If you see the ICMP "message too
> > long" packets arrive and they are not forwarded, the issue is on your
> > side, time to check the firewall.
> >
> > If you are 100% sure your side is right, it might be on the ISP side;
> > we actually had this issue until recently in our associative ISP and
> > we have seen large commercial providers have issues with this as well
> > (IPv6 not widely adopted around here).
> >
> > A workaround was to announce locally a lower MTU for IPv6 with
> > rtadvd(8).
> > Over time that proved to be an issue on Windows hosts, which didn't
> > seem to handle well a different MTU for IPv4 and IPv6, so I ended up
> > announcing the lower MTU internally for both stacks temporarily (using
> > DHCP for v4).
> >
> > Hope that helps and wasn't too all over the place, this is memory
> > talking here :-D.
> >
> > Cheers,
> > --
> > Evilham