>Number: 71965
>Category: kern
>Synopsis: TCP MSS issue in combination with ipfw fwd
>Confidential: no
>Severity: non-critical
>Priority: medium
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Tue Sep 21 10:00:43 GMT 2004
>Closed-Date:
>Last-Modified:
>Originator: Aragon Gouveia
>Release: 5.2.1-RELEASE-p9
>Organization:
>Environment:
FreeBSD gw.home.geek.sh 5.2.1-RELEASE-p9 FreeBSD 5.2.1-RELEASE-p9 #1: Sun Sep 12 15:11:33 SAST 2004 root at gw.home.geek.sh:/usr/obj/usr/src/sys/GW i386
>Description:
Below is a copy of my posts to freebsd-net as requested.
-----
Hi,
A while ago I setup a vtun tunnel between a FreeBSD 4.10-RELEASE machine and
a 5.2.1-RELEASE-p9 machine. Initially everything appeared to work great,
but I've just stumbled upon a seriously wierd problem that I can't figure
out.
I know this is not a support forum for the vtun package, but the problem I'm
having is consistently reproducable with another VPN type package - OpenVPN.
I'm beginning to think the problem is not related to vtun/OpenVPN and was
hoping someone could shed some light on my problem.
My setup is as follows.
My notebook is running 5.2.1-REALEASE-p9 using userland ppp to establish a
PPPoE session over an ADSL bridge. Above that, it runs vtun to establish a
UDP tunnel to my VPN server sitting at an ISP.
The VPN server is the 4.10-RELEASE machine also running vtun 1.6 configured
in server mode. The link is configured for no compression, but encryption
enabled.
Now the link comes up 100% and passes most data fine. I've set my interface
MTU down to 1200 on both sides. I'm also source routing on my notebook with
ipfw fwd to make sure traffic flows out the links I want them to flow. This
is working 100% too.
The problem is this: I'm running Apache 2.0.50 on my notebook. If I
request a page from my notebook from an outside machine via the VPN,
request responses that exceed the interface MTU are simply not sent.
For example, if I request a file sized at 1100 bytes, this is the tcpdump
transcript running on my notebook, sniffing the VPN interface:
tcpdump: listening on tun1
20:31:01.892060 <host>.3159 > <notebook>.80: S 4115956342:4115956342(0) win 57344 <mss 1460,nop,wscale 0,nop,nop,timestamp 606530904 0> (DF) [tos 0x10]
20:31:01.892228 <notebook>.80 > <host>.3159: S 1941194202:1941194202(0) ack 4115956343 win 65535 <mss 1452,nop,wscale 1,nop,nop,timestamp 269446043 606530904> (DF)20:31:01.974087 <host>.3159 > <notebook>.80: . ack 1 win 57600 <nop,nop,timestamp 606530914 269446043> (DF) [tos 0x10]
20:31:08.417478 <host>.3159 > <notebook>.80: P 1:21(20) ack 1 win 57600 <nop,nop,timestamp 606531558 269446043> (DF) [tos 0x10]
20:31:08.517285 <notebook>.80 > <host>.3159: . ack 21 win 33120 <nop,nop,timestamp 269452670 606531558> (DF)
20:31:10.340468 <host>.3159 > <notebook>.80: P 21:23(2) ack 1 win 57600 <nop,nop,timestamp 606531751 269452670> (DF) [tos 0x10]
20:31:10.341371 <notebook>.80 > <host>.3159: P 1:286(285) ack 23 win 33120 <nop,nop,timestamp 269454494 606531751> (DF)
20:31:10.341412 <notebook>.80 > <host>.3159: P 286:1386(1100) ack 23 win 33120 <nop,nop,timestamp 269454494 606531751> (DF)
20:31:10.342143 <notebook>.80 > <host>.3159: F 1386:1386(0) ack 23 win 33120 <nop,nop,timestamp 269454495 606531751> (DF)
20:31:10.568480 <host>.3159 > <notebook>.80: . ack 286 win 57600 <nop,nop,timestamp 606531773 269454494> (DF) [tos 0x10]
20:31:10.618594 <host>.3159 > <notebook>.80: . ack 1387 win 57600 <nop,nop,timestamp 606531779 269454494> (DF) [tos 0x10]
20:31:10.626417 <host>.3159 > <notebook>.80: F 23:23(0) ack 1387 win 57600 <nop,nop,timestamp 606531779 269454494> (DF) [tos 0x10]
20:31:10.626532 <notebook>.80 > <host>.3159: . ack 24 win 33119 <nop,nop,timestamp 269454779 606531779> (DF)
The two important lines being:
20:31:10.341371 <notebook>.80 > <host>.3159: P 1:286(285) ack 23 win 33120 <nop,nop,timestamp 269454494 606531751> (DF)
20:31:10.341412 <notebook>.80 > <host>.3159: P 286:1386(1100) ack 23 win 33120 <nop,nop,timestamp 269454494 606531751> (DF)
The first of these two is the HTTP response header, and the second the
actual requested data (1100 bytes as shown).
If I try request a file 1400 bytes large (MTU is 1200):
tcpdump: listening on tun1
20:35:06.588068 <host>.3161 > <notebook>.80: S 1461068997:1461068997(0) win 57344 <mss 1460,nop,wscale 0,nop,nop,timestamp 606555375 0> (DF) [tos 0x10]
20:35:06.588242 <notebook>.80 > <host>.3161: S 1654337904:1654337904(0) ack 1461068998 win 65535 <mss 1452,nop,wscale 1,nop,nop,timestamp 269690732 606555375> (DF)
20:35:06.659998 <host>.3161 > <notebook>.80: . ack 1 win 57600 <nop,nop,timestamp 606555381 269690732> (DF) [tos 0x10]
20:35:10.490089 <host>.3161 > <notebook>.80: P 1:21(20) ack 1 win 57600 <nop,nop,timestamp 606555765 269690732> (DF) [tos 0x10]
20:35:10.589617 <notebook>.80 > <host>.3161: . ack 21 win 33120 <nop,nop,timestamp 269694734 606555765> (DF)
20:35:11.506613 <host>.3161 > <notebook>.80: P 21:23(2) ack 1 win 57600 <nop,nop,timestamp 606555867 269694734> (DF) [tos 0x10]
20:35:11.507306 <notebook>.80 > <host>.3161: P 1:286(285) ack 23 win 33120 <nop,nop,timestamp 269695651 606555867> (DF)
20:35:11.716698 <host>.3161 > <notebook>.80: . ack 286 win 57600 <nop,nop,timestamp 606555888 269695651> (DF) [tos 0x10]
20:35:16.619379 <host>.3161 > <notebook>.80: F 23:23(0) ack 286 win 57600 <nop,nop,timestamp 606556378 269695651> (DF) [tos 0x10]
20:35:17.815936 <host>.3161 > <notebook>.80: F 23:23(0) ack 286 win 57600 <nop,nop,timestamp 606556498 269695651> (DF) [tos 0x10]
20:35:20.017123 <host>.3161 > <notebook>.80: F 23:23(0) ack 286 win 57600 <nop,nop,timestamp 606556718 269695651> (DF) [tos 0x10]
20:35:24.227404 <host>.3161 > <notebook>.80: F 23:23(0) ack 286 win 57600 <nop,nop,timestamp 606557138 269695651> (DF) [tos 0x10]
The two important lines being:
20:35:11.507306 <notebook>.80 > <host>.3161: P 1:286(285) ack 23 win 33120 <nop,nop,timestamp 269695651 606555867> (DF)
20:35:11.716698 <host>.3161 > <notebook>.80: . ack 286 win 57600 <nop,nop,timestamp 606555888 269695651> (DF) [tos 0x10]
The first line of the important ones is the HTTP response header. The
second one obviously just the TCP acknowledgement. The expected next few
packets that should follow the header never get sent.
I'm stumped. As I said, I've tried OpenVPN as well but the behaviour is
precisely the same. Does anyone have any ideas?
Thanks,
Aragon
----
Hi,
I've figured out the problem, but I'm not sure how to solve it. Here's what I
think is the problem.
>From a tcpdump transcript:
09:56:37.652907 <host>.4185 > <notebook>.80: S 487952620:487952620(0) win 57344 <mss 1460,nop,wscale 0,nop,nop,timestamp 6113642500> (DF) [tos 0x10]
09:56:37.653076 <notebook>.80 > <host>.4185: S 4069940133:4069940133(0) ack 487952621 win 65535 <mss 1452,nop,wscale 1,nop,nop,timestamp 317780676 611364250> (DF)
<notebook> is my notebook running Apache. As can be seen above, it's
negotiating an MSS of 1452 with the peer, which it should not be doing. The
reason it's doing that is because my default route is via an interface with
an MTU of 1492 - the tun interface opened by userland ppp for the PPPoE
session over my ADSL bridge.
As I said, I'm using ipfw fwd to source route packets from <notebook>
(the vtun tunnel interface address) to the vtun tunnel's remote end-point.
But I'm guessing MSS is chosen based on the host's routing table. Which
makes perfect sense.
So to prove my suspicion I added a route on my notebook as follows:
route add -host <host> 196.15.a.y
196.15.a.y being the vtun tunnel's remote end-point.
Now the tcpdump transcript looks like this:
10:10:21.227506 <host>.2404 > <notebook>.80: S 996010957:996010957(0) win 57344 <mss 1460,nop,wscale 0,nop,nop,timestamp 35602665280> (DF) [tos 0x10]
10:10:21.227717 <notebook>.80 > <host>.2404: S 2935622965:2935622965(0) ack 996010958 win 65535 <mss 1204,nop,wscale 1,nop,nop,timestamp 318604232 3560266528> (DF)
The tunnel's interface MTU was set at 1256 when I did this. So the
negotiated MSS is now correct and things are working.
But I need to be able to route based on source address and ipfw fwd is the
only way I know how to do it. Can anyone think of a clever workaround for
this? Is there a way to force the TCP stack to use a set MSS regardless of
what the routing table and interface MTU say?
Thanks,
Aragon
----
>How-To-Repeat:
>Fix:
Not sure exactly. If the TCP stack can see that ipfw is going to forward packets through an interface different to that specified by the routing table, it should be able to negotiate an appropriate MSS for the relevant session. I think the problem here is that TCP thinks the session will be routed out an interface that its getting from the host's routing table. I don't know exactly how all the internals work, so I'll stop talking now. :)
>Release-Note:
>Audit-Trail:
>Unformatted: