Web Server supporting up to 4 WANs/Interfaces

[Patrick Bihan-Faou]

Le 17/12/2010 08:15, Julian Elischer a écrit :
> On 12/16/10 10:32 PM, Jayster wrote:
>> I am in need of immediate help and while I rarely post on boards or 
>> lists (I promise I'll start sharing more of my own wisdom in the 
>> future), this issue has me stumped like no other. First of all, I 
>> need multiple gateways. Yes, I understand there is no such thing as 
>> multiple "default" gateways and I have a solid grasp of networking in 
>> general... . I also have a reasonable grasp of FreeBSD, though I am 
>> no where near the expert of many. I actually come from the world of 
>> programming and I've only used Linux and Windows in the past.
>>
>> I am currently working on a big project that involves many sites and 
>> a custom appliance with multiple applications designed in-house. In 
>> fact, I might be the ONLY person in snowy Michigan hiring right 
>> now... . After an exhaustive investigation, I chose FreeBSD over all 
>> other OSs. It was not a light choice, but I've been very pleased 
>> despite having zero knowledge at the start of it. The main reasons 
>> were flexibility, licensing and most important, I'm still shocked how 
>> open the community is to answering questions in posts, instead of 
>> putting people down for having less knowledge than a rude poster. I 
>> have found an answer to the most obscure questions through very quick 
>> google searches...until now. Kudos to those who take the 
>> time...FreeBSD far outnumbers other OS communities. Please help me 
>> avoid this issue being the big letdown, possibly forcing us to leave 
>> FreeBSD. I like the OS way too much already.
>>
>> Now the issue. Without too much detail, my device has 4 GigE ports on 
>> it. Each will be attached to a routed network. There is NO routing 
>> required between networks inside the box (not a router or firewall) 
>> and in fact, it CANNOT be allowed to happen because of security. 
>> Instead, each WAN port needs access to this box, but nothing beyond. 
>> The access consists of a Web Server, though several other Ports are 
>> required, such as SNMP Traps, Syslog, etc. Getting to the box is 
>> easy, routers do all the work. The issue is getting traffic back 
>> through the same interface it came in on and through the same router 
>> gateway. As we all know, only 1 gateway can be assigned in FreeBSD, 
>> unlike other flavors of Linux. Even the ones who don't offer single 
>> line gateway support can use IPTables to accomplish this task. But 
>> IPTables is not supported in FreeBSD. Not a bad thing as long as 
>> comparable solutions exist.
>>
>> Setting up static routes is not the solution. The problem with it is 
>> that multiple sites which will have this box will not have access to 
>> the next hop info from the gateway (the next hop gateway and subnet 
>> on the other side of the router). So I cannot use static routes.
>>
>> PFSense appears to support this (though not tested by me). I REALLY 
>> do not want to go that route. We have invested 3 months into adding 
>> many apps to the FreeBSD we have. PFSense is a custom FreeBSD kernel 
>> with many changes. Many message boards claim it breaks many Ports and 
>> changes other behaviors. Even if it didn't, we are under deadline and 
>> moving everything over to a new FreeBSD Version and then extensively 
>> testing everything repeatedly again would be a nightmare. I am 
>> interested in experiences with it if it becomes the last resort, though.
>>
>> I have tried both PF and IPFW. Different posts around the web claim 
>> Multiple Gateway solutions using both of them. I have tried each of 
>> the recommended setups, but had no luck. If you read the last 
>> responses to each of those posts, others also state they could not 
>> duplicate what is claimed, as well. PF looks the most promising. It 
>> has "if-bound", which is supposed to keep interface traffic on the 
>> same interface. That is a good first step. But pointing it to the 
>> gateway on that interface is still an issue. Please HELP!!! I haven't 
>> slept in days and I've been stuck for a week now!!! This is our last 
>> showstopper.
>>
>
> if you are running on freeBSD 8 or newer you have two solutions open 
> to you (maybe 3).
>
> * Firstly, you can assign a completely different routing table to each 
> socket so that
> packets from one socket only see things through the perspective of one 
> routing
> table but packets from another socket behave according to the rules of 
> a completely
> different routing table. alternatively you can assign a different 
> routing table to a
> process and its descendants.
> you can also use ipfw fwd to remap ports and addresses, in conjunction 
> with the
> different routing tables.  Routing tables are sometimes called FIBs 
> (Forwarding Information Bases)
>
> man setfib(1) and setfib(2) for more details.
> Also see the setfib socket option in setsockopt(2).
>
> There are details that are still open for development (like IPV6 
> support) but
> it sounds like it will do what you want.
>
> * The second option is the new jail support.
>
> while setfib and friends can easily allow a single process to act 
> differently on a socket by socket basis
> the new jail facilities allow you to take multiple interfaces and 
> assign them to different jails,
> and each jail can be given a completely different routing table or in 
> fact a completely different SET of routing tables.
>
> man jail   and man ifconfig            (ifconfig vnet)
>
> * As a poor third contender you can do really funky things with the 
> ipfw 'fwd' command.
>
> julian
>
> (let me know offline a bit more about what you want and maybe I can be 
> a bit more specific about how to do it.)

A third solution that does not involve jails or fib (which to me sounds 
like the better approach btw) is to use pf and "route-to" rules such as 
this one:


pass out quick on ! bge1 route-to (bge1 ip.of.gateway.on.bge1) from bge1 
to any

and repeat for all four network interface.

This should work properly for all TCP based protocols as the source IP 
on return packet from your server will be set to the IP of the interface 
the trafic came in from. The route-to mechanism will force the gateway 
and interface for the return packets based on that knowledge.

I use this on FreeBSD 6 based machines where the new setfib 
functionality is not available and that works like a charm.

I don't think any specific kernel option is needed for that. I tried the 
forward rules on ipfw, but they were not as easy to setup and not 
working reliably in my setup.

Best regards,

Patrick Bihan-Faou