Re: PPPoE (mpd5) IPv6 issues

From: Evilham <contact_at_evilham.com>
Date: Fri, 16 Sep 2022 20:00:23 UTC
Hey,

On dv., set. 16 2022, driesm@FreeBSD.org wrote:

> Hmm this mail was not finished, sorry about that.
>
> I will include the link that fell-off, IPv6 PPPoE MSS incorrect 
> | Netgate Forum
>
>
>
> Any help or pointers are greatly appreciated!
>
>
>
> PS: I use IPFW as my firewall, if this has anything to do with 
> it. I do have a “reassemble all in” rule at the top of my 
> ruleset
>
>
>
> From: driesm@FreeBSD.org <driesm@FreeBSD.org>
> Sent: Friday, 16 September 2022 18:36
> To: freebsd-net@freebsd.org
> Subject: PPPoE (mpd5) IPv6 issues
>
>
>
> Hi freebsd-net!
>
>
>
> After a lot of google and tinkering I have hit a brick wall 
> trying to set-up my new ISP which uses PPPoE.
>
> I’m at the point where IPv4 works fine 100% for all websites. 
> But from the moment I turn on IPv6 all sites that prefer IPv6 
> stop working, I
> have confirmed that routing table etc all look good. So I’m more 
> suspicious to a MTU / MSS issue.
>
> Although I cant find that much info about it. I have tcpmss fix 
> turned on in my mpd.conf
>
>
>
> There seems to be a few sites like google.com and youtube.com 
> that keep working.
>
>
>
> I have googled thoroughly but I was unable to find a fix for my 
> problem. The one close to my issue seems  this issue from 2020:


Some things to take into account are: MTU of the physical 
interface and ICMP blackholes, see below.

For the MTU of the physical interface, take into account that if a 
VLAN and PCP (vlan(4)+ifconfig(8)) are required by the wholesale 
provider, those consume 4 bytes each and taking PPPoE's 8 bytes 
into account as well, your physical interface should have a higher 
MTU (4 bytes for VLAN + 4 bytes for PCP + 8 bytes for PPPoE = 16 
bytes overhead on physical interface) than the connection (I'd 
guess one of 1500, 1492 or 1484 depending on the wholesale 
provider and the ISP).


Basically, from the router if the interface created by mpd5 has an 
MTU of 1500, you should be able to:

ping -6D -s 1452 ipv6.icmp.host

With -6D being "do not fragment"
And 1452 bytes being the payload =
  1500 bytes - 40 bytes from IPv6 headers - 8 bytes for ICMP 
  header


And you **should** receive a "ping: sendmsg: Message too long" 
with:

ping -6D -s 1453 ipv6.icmp.host


When in doubt, try:
ping -6D -s 1232 ipv6.icmp.host
This should work since 1280 bytes is the minimum MTU for IPv6 and 
gradually go up to establish what the maximum MTU for your 
connection is being.


If you don't receive the "Message too long" with way too large 
packets, there might be a bit of an ICMP blackhole on the ISP side 
or on the route to the ipv6.icmp.host.

This is usually not too tragic for IPv4 because routers do the 
fragmenting magic, but it is terrible for IPv6 since it is the 
client that has to fragment, and for that it needs to receive the 
"Message too long" ICMP response.

By using tcpdump on the router, convince yourself truly that it is 
not *you* dropping all incoming ICMP! If you see the ICMP "message 
too long" packets arrive and they are not forwarded, the issue is 
on your side, time to check the firewall.

If you are 100% sure your side is right, it might be on the ISP 
side; we actually had this issue until recently in our associative 
ISP and we have seen large commercial providers have issues with 
this as well (IPv6 not widely adopted around here).

A workaround was to announce locally a lower MTU for IPv6 with 
rtadvd(8).
Over time that proved to be an issue on Windows hosts, which 
didn't seem to handle well a different MTU for IPv4 and IPv6, so I 
ended up announcing the lower MTU internally for both stacks 
temporarily (using DHCP for v4).

Hope that helps and wasn't too all over the place, this is memory 
talking here :-D.

Cheers,
--
Evilham