IPSEC tunnel problem

Thu May 4 16:05:33 UTC 2006

Ok guys,
finally, problem appears to be PF's problem, not an IPSEC one.
We managed to build a test lab and reproduce failures.

At the moment we have:

+--------------------+
|                    |
| terminal server    | Windows box
| 10.2.0.2           |
+--------------------+
         |
         |
+--------------------+
| fxp1 10.2.0.1      |
|       gif0         | FreeBSD 6.0 <--- THIS BOX HAS PROBLEMS
| fxp0 10.1.0.2      |
+--------------------+
         |
   ipsec goes here
         |
+--------------------+
| fxp1 10.1.0.1      |
|       gif0         | FreeBSD 6.0
| fxp0 192.168.10.71 |
+--------------------+
         |
         |
+--------------------+
| 192.168.10.100     |
| Remote desktop     | Windows box
| client             |
+--------------------+

Everyting below applies to the FreeBSD box marked as "THIS BOX HAS
PROBLEMS" on the "picture" above.

gif0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1280
        tunnel inet 10.1.0.2 --> 10.1.0.1
        inet6 fe80::203:47ff:fe08:3fa6%gif0 prefixlen 64 scopeid 0x5 
        inet 10.2.0.1 --> 192.168.10.71 netmask 0xffffff00

when client connects to Remote Desktop server everything works for some
time and then hangs. On the fxp1 of the box I see:

...
19:52:53.658958 IP 10.2.0.2.3389 > 192.168.10.100.1919: .
62480:63446(966) ack 27922 win 64808
19:52:53.659790 IP 10.2.0.2.3389 > 192.168.10.100.1919: P
63446:64115(669) ack 27922 win 64808
19:52:53.661247 IP 192.168.10.100.1919 > 10.2.0.2.3389: . ack 64115 win
65535
19:52:53.768253 IP 10.2.0.2.3389 > 192.168.10.100.1919: .
64115:65081(966) ack 27922 win 64808
19:52:53.769090 IP 10.2.0.2.3389 > 192.168.10.100.1919: P
65081:65936(855) ack 27922 win 64808
19:52:53.769094 IP 10.2.0.2.3389 > 192.168.10.100.1919: .
65936:66902(966) ack 27922 win 64808
19:52:53.769097 IP 10.2.0.2.3389 > 192.168.10.100.1919: P
66902:67534(632) ack 27922 win 64808
19:52:53.769100 IP 10.2.0.2.3389 > 192.168.10.100.1919: .
67534:68500(966) ack 27922 win 64808
19:52:53.769103 IP 10.2.0.2.3389 > 192.168.10.100.1919: P
68500:68793(293) ack 27922 win 64808
19:52:53.771737 IP 10.2.0.1 > 10.2.0.2: ICMP host 192.168.10.100
unreachable, length 36
19:52:53.772730 IP 10.2.0.1 > 10.2.0.2: ICMP host 192.168.10.100
unreachable, length 36
19:52:53.773724 IP 10.2.0.1 > 10.2.0.2: ICMP host 192.168.10.100
unreachable, length 36

Note that at some moment box starts rejecting packets replying with ICMP
host unreach.
Below is output of pfctl -s info after the test. (Just before the test
started all the stats were reset):

Status: Enabled for 0 days 00:06:00             Debug: Misc

Hostid: 0x1000314c

State Table                          Total             Rate
  current entries                        3               
  searches                            1387            3.9/s
  inserts                                3            0.0/s
  removals                               0            0.0/s
Counters
  match                                  3            0.0/s
  bad-offset                             0            0.0/s
  fragment                               0            0.0/s
  short                                  0            0.0/s
  normalize                              0            0.0/s
  memory                                 0            0.0/s
  bad-timestamp                          0            0.0/s
  congestion                             0            0.0/s
  ip-option                              0            0.0/s
  proto-cksum                            0            0.0/s
  state-mismatch                        13            0.0/s
  state-insert                           0            0.0/s
  state-limit                            0            0.0/s
  src-limit                              0            0.0/s
  synproxy                               0            0.0/s

Note the state-mismatch counter. I have turned on debug for pf (pfctl -x
misc) and grabbed its output during that test. It outputs alot of
"kernel: pf: loose state match: TCP 10.2.0.2:3389 10.2.0.2:3389" which I
assume are normal. But then the following appears:

May  4 19:52:53 vrn1 kernel: pf: BAD state: TCP 10.2.0.2:3389
10.2.0.2:3389 192.168.10.100:1919 [lo=4162748520 high=4162681620
win=65535 modulator=0] [lo=0 high=65535 win=1 modulator=0] 2:0 PA
seq=4162748520 ack=0 len=632 ackskew=0 pkts=245:0 dir=out,fwd
May  4 19:52:53 vrn1 kernel: pf: State failure on: 1       | 5  
May  4 19:52:53 vrn1 kernel: pf: BAD state: TCP 10.2.0.2:3389
10.2.0.2:3389 192.168.10.100:1919 [lo=4162748520 high=4162681620
win=65535 modulator=0] [lo=0 high=65535 win=1 modulator=0] 2:0 A
seq=4162749152 ack=0 len=966 ackskew=0 pkts=245:0 dir=out,fwd
May  4 19:52:53 vrn1 kernel: pf: State failure on: 1       | 5  
May  4 19:52:53 vrn1 kernel: pf: BAD state: TCP 10.2.0.2:3389
10.2.0.2:3389 192.168.10.100:1919 [lo=4162748520 high=4162681620
win=65535 modulator=0] [lo=0 high=65535 win=1 modulator=0] 2:0 PA
seq=4162750118 ack=0 len=293 ackskew=0 pkts=245:0 dir=out,fwd
May  4 19:52:53 vrn1 kernel: pf: State failure on: 1       | 5 

I have no idea what this means....

# pfctl -s rules
No ALTQ support in kernel
ALTQ related functions disabled
pass in all keep state
pass out all keep state

If I use "pass in/out all" rules WITHOUT a "keep state" - everything
works fine.

I have tcpdump from all four interfaces (fxp0, fxp1, gif0, pflog0) as
well as full console output. Can send it to any party interested. Also,
for some time this lab will be available for any other tests.

Regards,
Dmitry Andrianov

________________________________

From: Dmitry Andrianov 
Sent: Friday, April 28, 2006 5:38 PM
To: freebsd-pf at freebsd.org
Subject: IPSEC tunnel problem

Hello.
First of all I apologize if I freebsd-pf is not the rigth place to ask
my question. I will explain below why it is actually asked here. But if
anyone knows the better place, let me know.
[stripped]