Jails and PF states on locahost

[Johan Ström]

Hello

I got a FreeBSD 6.2 box running a few jails, with a pretty strict PF  
ruleset. I got a problem with traffic between two of the jails. Both  
have public IPs (one of them have two using the jail-multiple-ip- 
patch). The problem I have is when they are to talk with each other.  
First let med describe the PF ruleset (somewhat stripped down but  
this should be the relevant stuff)

jail1=xx.xx.xx.131
jail2a=xx.xx.xx.133
jail2b=xx.xx.xx.134
scrub in all
block drop in log
# base system talk to itself
pass in on lo0 inet from 127.0.0.1 to 127.0.0.1

# all can talk out
pass out on em0 proto tcp flags S/SA modulate state
pass out on em0 proto udp keep state

# jails talk to them selfs
pass in on lo0 inet from $jail1 to $jail1
pass in on lo0 inet from {$jail2a $jail2b} to {$jail2a $jail2b}

# let smtp in on jail1
pass in on {lo0 em0} inet proto tcp from any to $jail1 port smtp  
flags S/SA modulate state

Okay, so the problem occurs when jail2 shall talk to jail1 on port 25  
(smtp). From the above rules, when the traffic leaves jail2 (traffic  
comes from $jail2b it seems) it should match the last rule and create  
a state. And so it does!

self tcp xx.xx.xx:25 <- xx.xx.xx.134:57557       SYN_SENT:ESTABLISHED
    [3014249759 + 65536](+2074393365) wscale 1  [4121000179 + 65536] 
(+541973245) wscale 1
    age 00:01:03, expires in 00:00:01, 7:10 pkts, 384:640 bytes

So the SYN arives at $jail1, but the SYNACK fails to go back to  
$jail2b (where the state should let the packet back in?), which is  
also seen in the following row from pflog0:

09:30:34.370402 rule 1/0(match): block in on lo0: (tos 0x0, ttl  64,  
id 35618, offset 0, flags [DF], proto: TCP (6), length: 64) xx.xx.xx. 
131.25 > xx.xx.xx.134.57557: S 793675827:793675827(0) ack 4121000179  
win 65535 <mss 1460,nop,wscale 1,[|tcp]>

So.. What have I missed? The state is created but it doesnt seem to  
match enough bytes or something? 384:640 matched packets, so et  
matches in both directions?

Any clues are welcome! Thanks

--
Johan Ström
Stromnet
johan at stromnet.se
http://www.stromnet.se/