TCP options order changed in FreeBSD 7, incompatible with some routers

[Bjoern A. Zeeb]

On Fri, 4 Apr 2008, Steven Hartland wrote:

Hi,

> I believe this has already been covered in quite some depth
> and iirc its regards the ordering of the new tcp flags
> introduced in 7. Best to have a look in the list archives for
> the specifics.

so as more users come and see this I am still trying to identify that
it's really the ordering of tcp options and not the bad padding.


History:

* the original problem showed up the first time

* people thought it was different option ordering

* tcp option ordering was changed back

* it was disocvered that there was bad padding after the options

* unfortunately changing back tcp options ordering hides the padding
   problem as the bad padding does not occur

* padding fix was comitted

* both changes have been MFCed to 7-STABLE.



What you can do:

I am talking to someone else but the more people with resources could
verify this the next days/weeks the more confident we can be.


If you are running 7.0-RELEASE please apply this patch:

1.141.2.4  +10 -2 src/sys/netinet/tcp_output.c  << that's the padding fix
http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/netinet/tcp_output.c.diff?r1=1.141.2.3;r2=1.141.2.4


rebuild your kernel, test with your users if things work ok now. If
they do (for all of them but the usual noise;) let us know.

If it does not help, apply the following patch as well and everything should be
fine (do not use TCP-MD5 as changing the order introduced another bug).

1.157.2.2  +5 -2 src/sys/netinet/tcp_var.h << that's the ordering fix
http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/netinet/tcp_var.h.diff?r1=1.157.2.1;r2=1.157.2.2


In case you do not have the resources either directly apply both
patches or go to 7-STABLE (and do not use TCP-MD5 either).


There are no binary updates for this yet.

> ----- Original Message ----- From: "s3raphi" <seraphi.lord at gmail.com>
>> 
>> I upgraded many web servers to FreeBSD 7.0-Release several weeks ago. These
>> servers serve hundreds of thousands of users. Since then, we have had many
>> users complain that they cannot connect to these servers any more. This was
>> a very tricky problem to diagnose, but using packet captures on both the
>> servers and the clients who have the problem I ended up with the same
>> results as the original poster. The user can ping the server with ICMP. The
>> user cannot complete a TCP connection.
>> Client sends SYN to server
>> Server responds SYN/ACK
>> Client packet capture does not show the SYN/ACK arrive.
>> Connection fails.
> ....


what would also be interesting to know is:

- Did the (SOHO) router drop it or the windows (XP) or another OS
   like MAC OSX, linux, or ...

- Which (SOHO) router - if you think this is the culprit - vendor/model

- If it was the windows, and it was XP, was the XP firewall on?

- If it was the windows, are there any other 3rd party firewalls
   running on it.




Someone may not understand why all these questions are important if
there is a patch available already:

a) finding out which change broke things is good for documentation
    purposes. Especially if it would be the tcp options ordering

b) if it wasn't options ordering we might want to go back to the
    more effective version



Bjoern


PS: you will also find tcpdump patches to show the tcp options and the
bad padding in this thread:

http://lists.freebsd.org/pipermail/freebsd-net/2008-March/017267.html


-- 
Bjoern A. Zeeb                                 bzeeb at Zabbadoz dot NeT
Software is harder than hardware  so better get it right the first time.