Re: IPv6 MTU discovery - packet too big

On Mon Dec 9, 2024 at 11:47 PM CET, Souji Thenria wrote:
> Hey all,
>
> On a VPS, I want to create separate jails for most services and assign
> each jail a public IPv6 address. However, I ran into an MTU issue, where
> the external interface of the host system sends multiple ICMPv6
> messages, stating that the received packets are too big to a remote
> server I tried to connect to from inside a jail. And the other server is
> ignoring these messages.
>
> I'm running FreeBSD 14.1-RELEASE on that server and use Bastille to
> manage my jails.
>
> The setup is as follows:
> <epair1> -- <eapir0> -- <bridge> -- <ext_inter>
>
> ext_inter: This interface is connected to the internet and has a public
> IPv6 address. It is NOT connected to the bridge.
>
> bridge: The bridge acts as default gateway for the jails and has a
> public IPv6 address assigned to it.
>
> epair0: Is a member of the bridge.
>
> epair1: This interface is passed to the jail, and a public IPv6 address is
> assigned inside the jail.
>
> The idea is that the jails can communicate over the bridge with each
> other, and when communicating with hosts on the internet, the traffic is
> routed over the ext_inter interface.
> All interfaces have an MTU of 1500 configured.
>
>
> The Problem:
> When I try to connect to, e.g. a web server, the ext_inter interface
> sends a lot of ICMPv6 packets saying:
> ICMP6, packet too big, mtu 1500, length 1240
>
> When I make the same request from the host itself, it works without any
> issues. I suspect that this is because the ext_inter interface has the
> 'JUMBO_MTU' option set, allowing packets to pass with a larger MTU.
> However, this shouldn't happen since the bridge and epair0/1 don't have
> this option.
>
> I can also confirm that the ICMP messages pass the firewall and reach
> the remote server. However, all servers I tried seemed to ignore that
> message and resent their packets without fragmenting them to a fitting
> size.
>
>
> Does anyone know what the issue might be, or have they had a similar
> problem and been able to solve it?
>
> Regards,
> Souji
>
>
> --
> Souji Thenria
> Website: www.souji-thenria.net

Hi all,

I finally found the Solution to my problem. In case anyone else runs
into it, if you are on a VPS (or any other virtual machine) and use the
vtnet driver, you just have to set the following in /boot/loader.conf:

hw.vtnet.csum_disable="1"
hw.vtnet.tso_disable="1"
hw.vtnet.lro_disable="1"

Only disabling checksum offloading might actually be enough.

Regards,
Souji

-- 
Souji Thenria
Website: www.souji-thenria.net