pf reply-to broken in RELENG_7

[Attila Nagy]

On 03/06/08 13:17, Max Laier wrote:
> Am Do, 6.03.2008, 09:36, schrieb Attila Nagy:
>   
>> Hello,
>>
>> I've just upgraded some of our 6-STABLE servers to 7-STABLE to notice
>> that pf reply-to for directly connected IPs seems to be broken.
>>
>> I have the following relevant rule in pf.conf:
>> pass in on $ext_if reply-to ( $ext_if csmvip ) proto tcp from any to any
>> port 25 label "mxtraffic-tcp" keep state
>>
>> which routes incoming SMTP connections (to be exact, the replies to
>> them) to the csmvip host, which is a load balancer. This is needed
>> because the LB doesn't do source NAT (it does destination NAT however to
>> direct traffic addressed to its virtual IP to the real servers' IPs),
>> and the servers have a different default route than the LB. This way the
>> servers reply to the LB, so it can rewrite the replies' source address
>> to its virtual IP, so the client will see the correct IP (the LB's
>> virtual IP) in the address, instead of the host's real address.
>>
>> It seems that this still works in 7-STABLE for the internet (not
>> directly connected) hosts, but not for directly connected hosts, for
>> example the ones, which are in the same subnet as my servers.
>> To overcome this, I've had to add static ARP entries to the servers, to
>> tell that the clients' hardware address is the address of the load
>> balancer, but it would be better if the previous behaviour (as in
>> 6-STABLE) could be restored.
>>
>> Could anybody help to resolve this?
>>     
>
> Might be the lack of sleep and coffee, but I can't quite figure out the
> network layout you are talking about.  Could you draw up a small example
> setup so I can follow?  Or at least (pseudo-)IP addresses for client,
> load-balancer, pf-box and servers?
>   
Of course, see: http://people.fsn.hu/~bra/freebsd/route-to-20080306/lb.png

10.0.0.1 is a normal router, which connects the servers to the internet 
(they can be reached at their real IPs through it and they can reach the 
internet via them). This is the default GW for the hosts.
10.0.0.2 is a load balancer, which has another address too (10.1.1.2), 
which is what I've called virtual IP. We do the service on this virtual 
IP, so the clients on the internet connect to this.
10.0.0.3 and 10.0.0.4 are the real servers behind the virtual IP.

The load balancer does this:
- a client -say 10.2.2.2 on the internet- connects to the load 
balancer's virtual IP, 10.1.1.2, for example with TCP/25
- the load balancer then takes a machine out of its configured pool for 
this service, for example 10.0.0.3
- to reach 10.0.0.3:25, the load balancer changes the packet's 
destination address from 10.1.1.2 to 10.0.0.3 and sends it out on its 
10.0.0.2 interface
- the host 10.0.0.3 gets this and sees that the source address is not 
directly connected, so it tries to send it via the default GW 
(10.0.0.1), which is of course not good, so we have a reply-to rule, 
which forces these replies back to 10.0.0.2
- the load balancer gets the reply, then changes its source IP to 
10.1.1.2 (the virtual IP for this service) and sends out to the internet

We can't send the reply to 10.0.0.1 (the default GW), because it won't 
do the source NAT and the load balancer will see only one way of the 
traffic, and it won't let it through (it has a state table, so for 
example a TCP session must be built up through it, it has to follow the 
states).
We also can't use the load balancer as a default GW, because it's not a 
router, it's pointless to route normal outgoing traffic through it 
(which is originated at the hosts), and the load balancer wouldn't allow 
this anyway, because it only handles configured services and everything 
must be in its state table.

This part works OK both with FreeBSD 6 and 7, the replies to the servers 
go to the load balancer.

What does not work is the scenario, when 10.0.0.4 tries to connect to 
10.1.1.2. You can easily see, that this is a problem, because 10.0.0.4 
sends the first packet through 10.0.0.1, which in turn forwards it to 
10.1.1.2 (the virtual IP of the load balancer).
The load balancer then changes the destination IP from 10.1.1.2 to 
10.0.0.3 (I really should draw three or more boxes, but please imagine 
that 10.0.0.4 won't connect to a service in which it is placed, so it 
won't happen that both the source and the destination is the same 
machine) and sends it through its 10.0.0.2 interface.
The host gets the packet and sees that the source is 10.0.0.4, which is 
directly connected to it, so it sends the answer directly and 10.0.0.3 
will stand confused (sent a request to 10.1.1.2, got the reply from 
10.0.0.3).
This is the second thing, which is the above reply-to is there: the 
machine must send the reply to 10.0.0.2, not to 10.0.0.4. The load 
balancer then will change the source address to 10.1.1.2 and send the 
reply to 10.0.0.4, so everything will be OK.

This is what works in 6.x, but not in 7.

I hope I could make this clear enough, if not, please ask.

Thanks,

-- 
Attila Nagy                                   e-mail: Attila.Nagy at fsn.hu
Free Software Network (FSN.HU)                 phone: +3630 306 6758
http://www.fsn.hu/