Problem with Path MTU Discovery

[Jeremie Le Hen]

Hi,

I set a gif(4)-based IPSec tunnel between my network and a friend's one.
This works pretty well, except that Path MTU Discovery doesn't work.

Quick draw:
          MTU: 1500       MTU:1280       MTU:1500
    Comp1 --------- Gate1 -------- Gate2 -----+-- Comp2
    RELENG_5        RELENG_4       RELENG_5   |   RELENG_5
                                              |
                                              +-- Comp3
                                                  RELENG_5

When Comp1 is accessing Comp2's httpd, the client will stall, waiting
for further data which never arrives.  Comp2 keeps trying to send
packets of 1500 bytes and does not decreases their size although Gate2
is sending it ICMP need-to-frag packets.
I wasn't able to understand the problem.

Sysctl net.inet.tcp.path_mtu_discovery is set to 1 on both computers.
There is no firewall.


Here is a network dump from the server :
%%%
    12:56:38.447645 192.168.4.80.80 > 192.168.1.222.60514: . 1:1449(1448) ack 189 win 33304 <nop,nop,timestamp 145151931 155308029> (DF) (ttl 64, id 24337, len 1500)
    12:56:38.448227 192.168.4.13 > 192.168.4.80: icmp: 192.168.1.222 unreachable - need to frag (DF) (ttl 64, id 4088, len 56)
    12:56:38.864776 192.168.4.80.80 > 192.168.1.222.60514: . 1:1449(1448) ack 189 win 33304 <nop,nop,timestamp 145151973 155308029> (DF) (ttl 64, id 24338, len 1500)
    12:56:38.865358 192.168.4.13 > 192.168.4.80: icmp: 192.168.1.222 unreachable - need to frag (DF) (ttl 64, id 4089, len 56)
    12:56:39.504765 192.168.4.80.80 > 192.168.1.222.60514: . 1:1449(1448) ack 189 win 33304 <nop,nop,timestamp 145152037 155308029> (DF) (ttl 64, id 24339, len 1500)
%%%

As you can see it doesn't take care of ICMP need-to-frag packets.
I can see the same behaviour if I mount a NFS volume with TCP.  This
will simply hangs.  This doesn't happen with an UDP mount because
packets get fragmented.


When I use scp(1) from Comp2 to fetch a reasonable big file from Comp1,
Comp1 tries to send some packet of 1500 bytes to, Gate1 sends it some
ICMP need-to-frag, and Comp1 successfully decreases packet size to
1280 bytes :
%%%
    [...]
    12:31:23.717870 IP (tos 0x8, ttl  64, id 14269, offset 0, flags [DF], length: 1500) 192.168.1.222.22 > 192.168.4.4.50458: . 1866:3314(1448) ack 1442 win 33304 <nop,nop,timestamp 155150893 14 4994746> 
    12:31:23.718428 IP (tos 0x0, ttl  64, id 12065, offset 0, flags [DF], length: 56) 192.168.1.1 > 192.168.1.222: icmp 36: 192.168.4.4 unreachable - need to frag (mtu 1280) for IP (tos 0x8, ttl 64, id 14269, offset 0, flags [DF], length: 1500) 192.168.1.222.22 > 192.168.4.4.50458: [|tcp]
    12:31:23.718489 IP (tos 0x8, ttl  64, id 54770, offset 0, flags [DF], length: 1280) 192.168.1.222.22 > 192.168.4.4.50458: . 1866:3094(1228) ack 1442 win 33304 <nop,nop,timestamp 155150893 14 4994746>
    [...]
%%%


FYI,
Comp1 network interface is em(4) and Comp2's one is xl(4).

Comp1:
FreeBSD obiwan.tataz.chchile.org 5.4-STABLE FreeBSD 5.4-STABLE #24: Sat Jun 25 12:52:32 CEST 2005     root at obiwan.tataz.chchile.org:/usr/src/sys/i386/compile/OBIWAN  i386

Comp2:
FreeBSD filer 5.4-STABLE FreeBSD 5.4-STABLE #11: Tue Jun 21 17:25:06 CEST 2005     root at filer:/usr/obj/usr/src/sys/BSD54 i386

Some sysctls differing between Comp1 and Comp2:
--- sysctl-a.Comp1      Wed Jul 13 14:50:34 2005
+++ sysctl-a.Comp2     Wed Jul 13 14:47:14 2005
-kern.osreldate: 504102
+kern.osreldate: 504101
-net.inet.ip.random_id: 1
+net.inet.ip.random_id: 0
-net.inet.tcp.hostcache.count: 9
+net.inet.tcp.hostcache.count: 3
-net.inet.tcp.reass.overflows: 165
+net.inet.tcp.reass.overflows: 0
-net.inet.udp.recvspace: 41600
+net.inet.udp.recvspace: 42080
-debug.mpsafenet: 1
+debug.mpsafenet: 0


I tried also tried to connect to Comp3, but the behaviour is the same.
Thus my guess is that Gate2 (RELENG_5) is sending bad ICMP need-to-frag
packets, while Gate1 (RELENG_4) is sending good ones, because all Comp*
are RELENG_5, and don't behavie in the same way.

Does anyone have an idea why Path MTU Discovery doesn't work on Comp2 ?
-- 
Jeremie Le Hen
< jeremie at le-hen dot org >< ttz at chchile dot org >