Two Networks on one System

[Jerome Herman]

On 21/06/2011 00:13, Jon Radel wrote:
>
> Can networks A and B talk to each other?  I suspect not, otherwise 
> things would be just working even if all traffic went to the primary's 
> gateway, but I just wanted to check that there wasn't something else 
> bad happening.
>
> On the assumption that A and B are completely disconnected, then the 
> only solution for this problem that I know of is to do policy-based 
> routing using the source address or interface to make routing 
> decisions, rather than using solely the destination address.
>
> This is actually relatively trivial to do using PF.
>
> pass in on nic_a reply-to ($nic_a $gw_a)
> pass in on nic_b reply-to ($nic_b $gw_b)
 From what I understand, there are two different ISP providing access to 
two different interfaces. In this case I am very concerned with all the 
bizarre things that a reply-to might trigger.
What I mean is that nothing guarantees that a distant address will 
access the box from the same interface every time.
I do not know what causes connection to be made on either nic_a or 
nic_b. Three things come to mind :
  - Multiple DNS resolution for an entry
- Different routing rewriting depending on the ISP
- An IP block being migrated from ISP A to ISP B with routes being 
updated while clients are connected

So depending on the client route, packets from a given IP address can 
land on either interface. Actually two clients nated behind the same 
public address might end up on both interfaces at the same time.
Even though your solution should work 99% of the time , it can lead to 
pretty strange behavior. I am not completely sure of how reply-to works, 
notably with keep state (and of course OpenBSD manuals on PF are down 
right now, at least from here). I remember attempting similar setups and 
having quite a lot of trouble with ICMP (especially RST for that matter).

I guess that in order to cover all solutions there would be need to know 
what is exactly happening. The most important thing would be to know if 
both IP addresses on the server are public, or if there are private with 
DMZ/Port routing/NAT etc.
If there is only a master DNS on the server, then I guess the worst 
thing that could happen would be strange timeouts and connection reset. 
But I there are data to be updated from the outside (DB, slave DNS, 
logs, mails etc.) things could turn out pretty badly.

Actually I think the rules should be following not only the IP, but also 
the flags and the ports. Keep-state should take care of this once the 
connection is properly initiated, but during handshake I do not see how 
to guarantee proper resolution.

Jerome Herman