Multiple Firewalls with ipfilter?

[Christopher Smith]

On Thursday, March 27, 2003, at 09:31  AM, Michael Richards wrote:

> The problem here is really 2 pronged:
> 1) I need some means of realising that the firewall just died and
> transparently switching over to the backup or load balancing the two
> so if one dies the other takes up the slack.

This is really easy.  Setup some sort of dedicated link between them 
(serial, UTP on its own port, wireless - we use UTP).

Use some sort of heartbeat script to detect when the other machine 
(and/or its interfaces) are up or down.  When the other machine goes 
down, have the backup box reconfigure its interfaces appropriately.  
All our scripts basically do is sit there banging away with a ping and 
if the other machine doesn't respond, it takes over the other machine's 
IPs.

We have a few redundant setups like this, and they can switch merrily 
between each machine and only lose one or two ping packets.  The catch 
is...

> 2) I need a means of syncing the state info so existing connections
> won't be torn down if they end up going through the other firewall.

This is really hard.

Our firewalls are ipfilter based.  ipfs(8) allows state tables to be 
saved and restored.  However, there are some major problems:

1.  While ipfs is saving the state tables, the state table is locked 
for writing *and reading*.  This effectively means your router stops 
routing for as long as it takes to save the state table (and even with 
only a modest number of states - 4000 or so - it takes a good second or 
two on a dual 1Ghz P3).
2.  The saved state table doesn't always reload correctly on the other 
machine (it often causes kernel panics when it reloads, or leaves the 
state table in such a way that no new states can be added) thus largely 
defeating the purpose of having a redundant firewall.
3.  When ipfs reloads the state table it completely overwrites any 
existing state table.  So, your failover machine can't be doing any 
other firewalling or routing.
4.  Any new states created since the last time the state table was 
saved will not be duplicated when it is reloaded.

I spent months fiddling around with periodically saving the state 
table, copying it to another machine and reloading it to get a kludgy 
form of stateful failover working but couldn't get it to work reliably. 
  Since I don't have the programming skills or knowledge to modify 
IPFilter to do it "properly" I am waiting for someone else to do so.


Darryl is apparently working on state table syncing with IPFilter 4.0.  
This should (at the very least) allow machines to be setup in a 
hot-spare style arrangement with all states added to the table on one 
machine also added on the other (via a dedicated link).  So the initial 
method of just reconfiuguring the interfaces on the fly should work 
fine.  Bear in mind, however, this has been in the works for at least a 
year.  Ideally it will allow selective addition and removal of state 
table entries and the ability to sync state tables between multiple 
machines.

I imagine there are people working on pf (OpenBSD) trying to do this 
sort of thing as well.

I have no idea if the people working on ipfw are trying to implement 
such a system.

If I had to make a prediction, I'd say the OpenBSD guys will get there 
first with pf.  If they do, they'll really have a killer app in the 
firewalling market.

> Sounds like a solution people would normally pay an obscene amount of
> money for but I'd be surprised if there isn't a way to do this. Maybe
> something with routing could do the balancing...

Yes, stateful failover does cost obscene amounts of money (AU$50k + for 
a Cisco solution - and that's with a discount).

The only real problem involved is synchronising the state tables 
between machines.

-- 
+- Christopher Smith, Systems Administrator 
------------------------------+
|  Server & Security Group, Information Technology Services             
   |
|  The University of Queensland, Brisbane, Australia, 4072              
   |
+- Ph +61 7 3365 4046 | email csmith at its.uq.edu.au | Fax +61 7 3365 
4065 -+