IPSEC tunnel problem

Fri May 5 12:07:18 UTC 2006

Ok, finally,  IPSEC_FILTERGIF fixes the problem with stalling
connections.

Daniel, thanks a lot for your help.

However I see at least three issues which needs to be handled somehow:

1. I believe that these days FreeBSD+pf becomes very popular
configuration and the fact that system does not work out of the box in
this configuration and requires some tricks is not a good sign. My
personal opinion is that IPSEC_FILTERGIF just should be on by default.
Or at least IPSEC section of FreeBSD handbook should say something about
it. Making kernel option on by default has advantage because there are
many places with IPSEC how-to on Internet and it is not possible to fix
all of them. So the question is should I fill doc/* PR or kernel/* PR
for that?

2. Why the router replies with ICMP host-unreachable to the TCP packets
with wrong state if block policy is set to "drop" and not "return"?

3. Below is the dump of packets how pf sees 3way handshake with
IPSEC_FILTERGIF option

15:22:53.192253 rule 0/0(match): pass in on fxp0: 10.1.0.1 > 10.1.0.2:
ESP(spi=0x00002708,seq=0x9b05), length 104
15:22:53.192267 rule 0/0(match): pass in on fxp0: 10.1.0.1 > 10.1.0.2:
192.168.10.176.2169 > 10.2.0.2.3389: S 327999972:327999972(0) win 32768
<mss 1460,nop,nop,timestamp 0 0,nop,nop,sackOK> (ipip-proto-4)
15:22:53.192276 rule 0/0(match): pass in on gif0: 192.168.10.176.2169 >
10.2.0.2.3389: S 327999972:327999972(0) win 32768 <mss
1460,nop,nop,timestamp 0 0,nop,nop,sackOK>
15:22:53.192286 rule 0/0(match): pass out on fxp1: 192.168.10.176.2169 >
10.2.0.2.3389: S 327999972:327999972(0) win 32768 <mss
1460,nop,nop,timestamp 0 0,nop,nop,sackOK>

15:22:53.192436 rule 0/0(match): pass in on fxp1: 10.2.0.2.3389 >
192.168.10.176.2169: S 1559792967:1559792967(0) ack 327999973 win 65535
<mss 1460,nop,nop,timestamp 0 0,nop,nop,sackOK>
15:22:53.192445 rule 0/0(match): pass out on gif0: 10.2.0.2.3389 >
192.168.10.176.2169: S 1559792967:1559792967(0) ack 327999973 win 65535
<mss 1460,nop,nop,timestamp 0 0,nop,nop,sackOK>
15:22:53.192460 rule 0/0(match): pass out on fxp0: 10.1.0.2 > 10.1.0.1:
ESP(spi=0x00002707,seq=0x18a4), length 104

15:22:53.193098 rule 0/0(match): pass in on fxp0: 10.1.0.1 > 10.1.0.2:
ESP(spi=0x00002708,seq=0x9b06), length 96
15:22:53.193108 rule 0/0(match): pass in on fxp0: 10.1.0.1 > 10.1.0.2:
192.168.10.176.2169 > 10.2.0.2.3389: . ack 1 win 33580
<nop,nop,timestamp 18151724 0> (ipip-proto-4)
15:22:53.193116 rule 0/0(match): pass in on gif0: 192.168.10.176.2169 >
10.2.0.2.3389: . ack 1 win 33580 <nop,nop,timestamp 18151724 0>
15:22:53.193123 rule 0/0(match): pass out on fxp1: 192.168.10.176.2169 >
10.2.0.2.3389: . ack 1 win 33580 <nop,nop,timestamp 18151724 0>

Clearly, pf sees each packet coming from the tunnel as four (ESP in on
WAN interface, ipencap in on WAN interface, in on gif and out packet on
LAN interface) while only three packets for data coming from LAN to
tunnel.

For me this has two disadvantages:
* I have to allow ipencap between endpoint in firewall, otherwise
decapulated IPSEC packet gets blocked. I do not like this thing because
it will also allow someone sending ipencap packets with spoofed source
easily.
* It is just a bit inconsistent - I would prefer either seeing none of
ipencaps (in and out) or seeing them both.

I could fill PR about that one too, but I'm not sure what is the best
way to propose.

Regards,
Dmitry Andrianov

-----Original Message-----
From: Dmitry Andrianov 
Sent: Friday, May 05, 2006 1:42 AM
To: 'Daniel Hartmeier'
Cc: freebsd-pf at freebsd.org
Subject: RE: IPSEC tunnel problem

Daniel,
Looks like you are right. Below is dump from pflog0 interface of 3way
handshake to 3389:

01:27:55.888031 rule 0/0(match): pass out on fxp1: 192.168.10.176.3809 >
10.2.0.2.3389: S 220200183:220200183(0) win 32768 <mss
1460,nop,nop,timestamp[|tcp]>
01:27:55.888177 rule 0/0(match): pass in on fxp1: 10.2.0.2.3389 >
192.168.10.176.3809: S 1690303870:1690303870(0) ack 220200184 win 65535
<mss 1460,nop,nop,timestamp[|tcp]>
01:27:55.888186 rule 0/0(match): pass out on gif0: 10.2.0.2.3389 >
192.168.10.176.3809: S 1690303870:1690303870(0) ack 220200184 win 65535
<mss 1460,nop,nop,timestamp[|tcp]>
01:27:55.888884 rule 0/0(match): pass out on fxp1: 192.168.10.176.3809 >
10.2.0.2.3389: . ack 1 win 33580 <nop,nop,timestamp 17650657[|tcp]>

Packets coming from the VPN only appear as "out" packets on LAN
ethernet. While packets coming from LAN appear as "in" packets on
ethernet and "out" packets on gif.

I assume this is because my kernel was not build with 

# Set IPSEC_FILTERGIF to force packets coming through a gif tunnel # to
be processed by any configured packet filtering (ipfw, ipf).
# The default is that packets coming from a tunnel are _not_ processed;
# they are assumed trusted.
#
# IPSEC history is preserved for such packets, and can be filtered #
using ipfw(8)'s 'ipsec' keyword, when this option is enabled.
#
#options        IPSEC_FILTERGIF         #filter ipsec packets from a
tunnel

Well. I knew about this option before but I thought it makes sense only
when I want filtering packets on gif0 which I currently do not. I never
realized absence of that option may break something else as happened in
my case.

Tomorrow will rebuild kernel and re-run my tests.

To me things like that are subject for FAQ or even better - should be on
by default. IMHO.

Regards,
Dmitry Andrianov

-----Original Message-----
From: Daniel Hartmeier [mailto:daniel at benzedrine.cx]
Sent: Thursday, May 04, 2006 8:40 PM
To: Dmitry Andrianov
Cc: freebsd-pf at freebsd.org
Subject: Re: IPSEC tunnel problem

On Thu, May 04, 2006 at 08:03:55PM +0400, Dmitry Andrianov wrote:

> May  4 19:52:53 vrn1 kernel: pf: BAD state: TCP 10.2.0.2:3389
> 10.2.0.2:3389 192.168.10.100:1919 [lo=4162748520 high=4162681620
> win=65535 modulator=0] [lo=0 high=65535 win=1 modulator=0] 2:0 PA 
> seq=4162748520 ack=0 len=632 ackskew=0 pkts=245:0 dir=out,fwd

The 'dir=out,fwd' part means that the state was created from a packet
going out on the interface (gif0, I assume), and that the packet being
blocked here was in the same direction.

The 'pkts=245:0' part means that the state entry has so far matched 245
packets flowing in the same direction (out), but 0 in the reverse
direction (in).

And that's the problem, pf is not associating replies with the state
entry. Because of that, the state entry does not advance its sequence
number window (advertised in the replies), and eventually stalls the
connection.

This is probably related to the gif interface. I haven't tried it on
FreeBSD, but for stateful filtering, it would be important that pf sees
packets in both directions on that interface (i.e. SYN outgoing, SYN+ACK
incoming, etc.)

You can test what packets pf sees in what direction on an interface by
replacing the ruleset with a single rule like

  pass log all

and observing pflog0 (with tcpdump, for instance) while establishing a
connection. If only packets of one direction are seen (or both outgoing
and incoming packets are seen as having the same direction), there might
be a problem with pfil hooks in gif.

Daniel