bin/118005: Can No Longer SSH into 7.0 host

[Andre Oppermann]

Hi Rob,

> Since upgrading to 7.0 I am no longer able to SSH into my server. I
> cvsup'ed to 7.0 code and rebuild world and since then I have had this
> issue. I have rebuilt multiple times in beta 1, 1.5 and 2. I can SSH into
> my host from some hosts within the local LAN. Some machines from outside my
> LAN I cannot ssh into this host. Hosts on my lan I have ssh'ed into this
> host with are windows(putty), Linux, and Solaris. From outside my LAN I
> cannot ssh into my host from Freebsd 6.2, Openbsd 4.1, and Linux(RHEL 4U4).
> Freebsd & Openbsd machines are on my home network. However my OSX laptop and
> windows machine, from my home network, can SSH into the host without a
> problem.

This is very strange as the operating systems you cite here correctly
implement timestamps and pass all tests that were added to FreeBSD
7BETA.

Please see the comments below for an analysis of what may be going on
here.

> voltron# tcpdump -e -vvnn port 22 and host 192.168.3.132
> tcpdump: listening on em0, link-type EN10MB (Ethernet), capture size 68
> bytes

A capture size of 68 bytes is insufficient to see all TCP headers.
Please increase it to 100 bytes with "-s 100".

> 08:09:55.816411 00:90:5f:0c:00:00 > 00:18:fe:67:54:76, ethertype IPv4
> (0x0800), length 74: (tos 0x0, ttl 61, id 56887, offset 0, flags [DF], proto
> TCP (6), length 60) 192.168.3.132.41922 > 192.168.8.163.22: S
> 722288481:722288481(0) win 5840 <mss 1460,sackOK,timestamp 1350033750[|tcp]>

The client sends the SYN to the server including timestamps (full
information cut off from too small capture size).

> 08:09:55.816432 00:18:fe:67:54:76 > 00:00:0c:07:ac:09, ethertype IPv4
> (0x0800), length 74: (tos 0x0, ttl 64, id 27230, offset 0, flags [DF], proto
> TCP (6), length 60) 192.168.8.163.22 > 192.168.3.132.41922: S
> 2406244836:2406244836(0) ack 722288482 win 65535 <mss 1460,nop,wscale
> 3,nop,nop,timestamp[|tcp]>

Server answers SYN with SYN|ACK, mss, window scale and timestamps.
Everything perfectly fine so far.

> 08:09:55.816925 00:90:5f:0c:00:00 > 00:18:fe:67:54:76, ethertype IPv4
> (0x0800), length 60: (tos 0x0, ttl 58, id 0, offset 0, flags [none], proto
> TCP (6), length 40) 192.168.3.132.41922 > 192.168.8.163.22: ., cksum 0x6872
> (correct), 1:1(0) ack 1 win 0

This packet is completely broken.  It appears to come from the client
and to ACK the SYN|ACK from above.  However it advertises a zero window,
has a different TTL (58 vs. 61 for the other packets from the client),
a IPID of zero and contains no timestamps.  This causes the check for
missing timestamps to trigger and to abort the connection.  RFC1323
requires timestamps to be always present when negotiated during the SYN
phase.

I've verified that FreeBSD 6.2, OpenBSD 4.1 and Linux RHEL4 implement
timestamps correctly and do not turn them off once negotiated.

This packet differs significantly from any other packet the client
sends.  It appears not to come from the real client but some other
device that either is in the path between client and server or to
share the IP address with the client.  Do you have any routers,
firewalls, gateways or NAT devices along the path that may cause
this strange packet to be sent?

> 08:09:55.816933 00:18:fe:67:54:76 > 00:00:0c:07:ac:09, ethertype IPv4
> (0x0800), length 54: (tos 0x0, ttl 64, id 27231, offset 0, flags [DF], proto
> TCP (6), length 40) 192.168.8.163.22 > 192.168.3.132.41922: R, cksum 0x47e3
> (incorrect (-> 0xd2ed), 2406244837:2406244837(0) win 0

The server sends the reset and aborts the connection because the
timestamp check failed.

> 08:09:55.817215 00:90:5f:0c:00:00 > 00:18:fe:67:54:76, ethertype IPv4
> (0x0800), length 66: (tos 0x0, ttl 61, id 56889, offset 0, flags [DF], proto
> TCP (6), length 52) 192.168.3.132.41922 > 192.168.8.163.22: ., cksum 0x8036
> (correct), 1:1(0) ack 1 win 1460 <nop,nop,timestamp 1350033751 1692996280>

Here the real client appears to ACK the SYN|ACK.  Note the correct TTL,
IPID, window size and timestamps.

> 08:09:55.833093 00:18:fe:67:54:76 > 00:00:0c:07:ac:09, ethertype IPv4
> (0x0800), length 105: (tos 0x0, ttl 64, id 27232, offset 0, flags [DF],
> proto TCP (6), length 91) 192.168.8.163.22 > 192.168.3.132.41922: P 1:40(39)
> ack 1 win 8326 <nop,nop,timestamp 1692996295 1350033751>

The server re-opens the connection based on the correct ACK because the
SYN cookie is correct.  The original syncache full entry was deleted
above when the reset was sent.  However the syncookie is still valid
and passed all tests.

> 08:09:55.833929 00:90:5f:0c:00:00 > 00:18:fe:67:54:76, ethertype IPv4
> (0x0800), length 60: (tos 0x0, ttl 61, id 8446, offset 0, flags [DF], proto
> TCP (6), length 40) 192.168.3.132.41922 > 192.168.8.163.22: R, cksum 0x59d0
> (correct), 722288482:722288482(0) win 0

The client answers with a reset because its socket was shut down due to
the reset above caused by the broken (spoofed) ACK packet earlier.


It all boils down to where the broken/spoofed ACK (packet #3) comes
from.  There must be some strange or completely broken device in the
packet path between client and server.  Please investigate and identify
potential candidates.

-- 
Andre