Strange behavior of MTU on loopback interfaces.

[Nikos Vassiliadis]

On 2/25/2011 9:29 AM, c0re wrote:
> Hello all!
>
> I'm testing setting lower MTU on loopback interfaces to avoid some MTU
> problems with IPSEC in a path of traffic.
>
> ifconfig lo1 create
> ifconfig lo1 mtu 1300
> ifconfig lo1 5.5.5.5/32
>
> # ifconfig lo1
> lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST>  metric 0 mtu 1300
>          inet 5.5.5.5 netmask 0xffffffff
>
> #ifconfig em0
> em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST>  metric 0 mtu 1500
>          options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
>          ether 12:ac:29:7c:fa:39
>          inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
>          media: Ethernet autoselect (1000baseTX<full-duplex>)
>          status: active
>
>
> And I set only one "Listen 5.5.5.5:80" in http.conf in apache 2.2
>
> # sockstat -4 | grep 80
> www      httpd      96843 3  tcp4   5.5.5.5:80            *:*
> www      httpd      96838 3  tcp4   5.5.5.5:80            *:*
> www      httpd      96837 3  tcp4   5.5.5.5:80            *:*
> www      httpd      96836 3  tcp4   5.5.5.5:80            *:*
> www      httpd      96835 3  tcp4   5.5.5.5:80            *:*
> www      httpd      96834 3  tcp4   5.5.5.5:80            *:*
> root     httpd      96833 3  tcp4   5.5.5.5:80            *:*
>
> I run tcpdump -ni em0 port 80. And made telnet 5.5.5.5 80 from other
> host and saw something wrong.
>
> 10:26:01.640866 IP 10.0.0.2.57553>  5.5.5.5.80: S
> 1049284626:1049284626(0) win 65535<mss 1460,sackOK,eol>
> 10:26:01.640902 IP 5.5.5.5.80>  10.0.0.2.57553: S
> 2144222949:2144222949(0) ack 1049284627 win 65535<mss
> 1460,sackOK,eol>
> 10:26:01.642632 IP 10.0.0.2.57553>  5.5.5.5.80: . ack 1 win 65535
>
> 5.5.5.5:80 said that it has got tcp mss 1460. Why? I was waiting for
> something like 1260.

It uses the MTU of the outgoing path, which is 1500.
You change the MTU for specific paths, using route and the mtu modifier.
Like this:

> lab# ifconfig em0
> em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>         options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
>         ether 08:00:27:17:c3:de
>         inet 192.168.73.193 netmask 0xffffff00 broadcast 192.168.73.255
>         media: Ethernet autoselect (1000baseT <full-duplex>)
>         status: active
> lab# route change 192.168.73.0 -mtu 1100
> change net 192.168.73.0
> lab# route -n get 192.168.73.0
>    route to: 192.168.73.0
> destination: 192.168.73.0
>        mask: 255.255.255.0
>   interface: em0
>       flags: <UP,DONE,STATIC>
>  recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
>        0         0         0         0      1100         1         0
> lab#

All packets going to 192.168.73.0/24 will use IP packet sizes up to
1100. IMHO it's better to leave the physical interface's MTU unchanged
and use the routing subsystem to define the maximum IP packet size per
path.

HTH, Nikos