Strange behavior of MTU on loopback interfaces.

[c0re]

2011/2/25 Nikos Vassiliadis <nvass at gmx.com>:
> On 2/25/2011 9:29 AM, c0re wrote:
>>
>> Hello all!
>>
>> I'm testing setting lower MTU on loopback interfaces to avoid some MTU
>> problems with IPSEC in a path of traffic.
>>
>> ifconfig lo1 create
>> ifconfig lo1 mtu 1300
>> ifconfig lo1 5.5.5.5/32
>>
>> # ifconfig lo1
>> lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST>  metric 0 mtu 1300
>>         inet 5.5.5.5 netmask 0xffffffff
>>
>> #ifconfig em0
>> em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST>  metric 0 mtu 1500
>>         options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
>>         ether 12:ac:29:7c:fa:39
>>         inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
>>         media: Ethernet autoselect (1000baseTX<full-duplex>)
>>         status: active
>>
>>
>> And I set only one "Listen 5.5.5.5:80" in http.conf in apache 2.2
>>
>> # sockstat -4 | grep 80
>> www      httpd      96843 3  tcp4   5.5.5.5:80            *:*
>> www      httpd      96838 3  tcp4   5.5.5.5:80            *:*
>> www      httpd      96837 3  tcp4   5.5.5.5:80            *:*
>> www      httpd      96836 3  tcp4   5.5.5.5:80            *:*
>> www      httpd      96835 3  tcp4   5.5.5.5:80            *:*
>> www      httpd      96834 3  tcp4   5.5.5.5:80            *:*
>> root     httpd      96833 3  tcp4   5.5.5.5:80            *:*
>>
>> I run tcpdump -ni em0 port 80. And made telnet 5.5.5.5 80 from other
>> host and saw something wrong.
>>
>> 10:26:01.640866 IP 10.0.0.2.57553>  5.5.5.5.80: S
>> 1049284626:1049284626(0) win 65535<mss 1460,sackOK,eol>
>> 10:26:01.640902 IP 5.5.5.5.80>  10.0.0.2.57553: S
>> 2144222949:2144222949(0) ack 1049284627 win 65535<mss
>> 1460,sackOK,eol>
>> 10:26:01.642632 IP 10.0.0.2.57553>  5.5.5.5.80: . ack 1 win 65535
>>
>> 5.5.5.5:80 said that it has got tcp mss 1460. Why? I was waiting for
>> something like 1260.
>
> It uses the MTU of the outgoing path, which is 1500.
> You change the MTU for specific paths, using route and the mtu modifier.
> Like this:
>
>> lab# ifconfig em0
>> em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>>        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
>>        ether 08:00:27:17:c3:de
>>        inet 192.168.73.193 netmask 0xffffff00 broadcast 192.168.73.255
>>        media: Ethernet autoselect (1000baseT <full-duplex>)
>>        status: active
>> lab# route change 192.168.73.0 -mtu 1100
>> change net 192.168.73.0
>> lab# route -n get 192.168.73.0
>>   route to: 192.168.73.0
>> destination: 192.168.73.0
>>       mask: 255.255.255.0
>>  interface: em0
>>      flags: <UP,DONE,STATIC>
>>  recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
>>       0         0         0         0      1100         1         0
>> lab#
>
> All packets going to 192.168.73.0/24 will use IP packet sizes up to
> 1100. IMHO it's better to leave the physical interface's MTU unchanged
> and use the routing subsystem to define the maximum IP packet size per
> path.
>
> HTH, Nikos
>

Works like a charm!

# route change 0.0.0.0 -mtu 1300
change net 0.0.0.0

# tcpdump -ni em0 host 5.5.5.5
13:42:58.996721 IP 10.0.0.2.51933 > 5.5.5.5.80: S
626695541:626695541(0) win 64512 <mss 1460,nop,nop,sackOK>
13:42:58.996760 IP 5.5.5.5.80 > 10.0.0.2.51933: S
289198669:289198669(0) ack 626695542 win 65535 <mss 1260,sackOK,eol>
13:42:58.999455 IP 10.0.0.2.51933 > 5.5.5.5.80: . ack 1 win 64512

Thank you very much!