Fwd: pf problems

[Sergey Lapin]

> Here's more proper description of the problem:
> 
> Hi.
> 
> While trying to configure pf on our freebsd 5.4 router we have
> encountered three bugs, one of which is minor, second is more serious.
> And the worst of all, because of the second bug we have to use
> workaround and that workaround triggers third bug which is actually
> critical (because it hangs device and floods the network). I'll try to
> explain these bugs briefly. Please advice where to report each fo
> them. We haev no idea if they are PF global or FreeBSD specific. We
> are going to repeat all the scenarios on OpenBSD.
> 
> Background: we have two ISP connections and it is the source of all
> our problems - looks like work in such a mode is not debugged very
> well in pf.
> 
> Setup:
>     Let's say ISP1 gives us 1.0.0.0/24 block with their gateway at
> 1.0.0.1. Our router is at 1.0.0.254.
>     Similarly, ISP2 gives us 2.0.0.0/24, their gateway is at 2.0.0.1
> and our router is at 2.0.0.254.
>     ISP2 gateway (2.0.0.1) is our default gateway.
>     We have DMZ where hosts of both ISP network live
> (dmz_net1=1.0.0.128/25 and dmz_net2=2.0.0.128/25).
> 
> 
>                    +---------------+
>          1.0.0.254 |               |
> -------------------+ ext_if1       |
>                    |               |  dmz_net1
>                    |        dmz_if +--------
>          2.0.0.254 |               |  dmz_net2
> -------------------+ ext_if2       |
>                    |               |
>                    +---------------+
> 
> 
> Note that firewall rules below are oversimplified to make rule listing
> compact. In our "real" firewall we differentiate between TCP and
> UDP/ICMP traffic to make sure all outgoing TCP connections use
> "modulate state" insted of "keep state" etc. Also ruleset below
> loosened in terms it uses "any" where our real ruleset explicitly
> defines networks. Again, this is only to demostrate problem with
> minimum amount of rules.
> 
> 
> ****************************************************
> ****************** BUG#1 (minor) *******************
> ****************************************************
> 
> When pf blocks incoming packet with "block return" rule, it does not
> return RST or ICMP packet to the interface from which original packet
> came from but always use default gateway instead. This way if we have
> default gateway set to ISP2's 2.0.0.1 and packet destined to 1.0.0.254
> comes from ISP1 interface (ext_if1) and this packet gets blocked with
> "block return", the TCP RST packet with source address 1.0.0.254 will
> be sent through 2.0.0.1 gateway. Obviously, ISP2 drops packets which
> source does not belong to their network so basically "block return"
> does not work at all.
> 
> 
> ****************************************************
> ****************** Bug#2 (severe) ******************
> ****************************************************
> 
> ruleset:
> 
> #########################################################33
> 
> #  nat outgoing connections on each internet interface
> nat on $ext_if1 from { $dmz_net2 } to any -> ($ext_if1)
> nat on $ext_if2 from { $dmz_net1 } to any -> ($ext_if2)
> 
> #  default deny silently
> block drop all
> 
> #  pass in quick any packets destined for the gateway itself
> pass in quick on $dmz_if from any to $dmz_if keep state
> 
> pass quick on lo0
> 
> #  Classify traffic from DMZ
> #  Allow all outgoing connections from DMZ
> 
> pass in on $dmz_if inet from $dmz_net1 to any keep state tag DMZ_TO_EXT1
> pass in on $dmz_if inet from $dmz_net2 to any keep state tag DMZ_TO_EXT2
> 
> #  Allow gateway to route between different networks on the DMZ
> pass in on $dmz_if inet from { $dmz_net1, $dmz_net2 } to { $dmz_net1,
> $dmz_net2 } keep state tag DMZ_TO_DMZ
> 
> #  Reroute OUT traffic appropriately
> pass out quick on $ext_if1 route-to ($ext_if2 $ext_gw2) tagged
> DMZ_TO_EXT2 keep state
> pass out quick on $ext_if2 route-to ($ext_if1 $ext_gw1) tagged
> DMZ_TO_EXT1 keep state
> 
> #  general "pass out" rules for external interfaces
> pass out on { $ext_if1, $ext_if2, $dmz_if } from any to any keep state
> 
> #########################################################33
> 
> Note that route-to rules should re-route all the traffic to proper
> interface regardless of default gateway specified.
> 
> What happens with such a rules:
> * When a host from dmz_net1 sends packets to the internet and default
> gateway is ISP1 one, everything works fine and no translation is
> performed.
> * When we change default gateway to the ISP2 very interesting thing
> happens: packet from dmz_net1 host leaves ext_if1 BUT it gets
> translated like it should leave ext_if2!
> 
> I do not understand this one completely.
> 
> 
> ****************************************************
> ***************** Bug#3 (critical) *****************
> ****************************************************
> 
> Because route-to for "out" rules did not work well, we decided to add
> the redirection for "in" rules so we added
> 
> pass in quick on $dmz_if route-to ($ext_if1 $ext_gw1) tagged
> DMZ_TO_EXT1 keep state
> pass in quick on $dmz_if route-to ($ext_if2 $ext_gw2) tagged
> DMZ_TO_EXT2 keep state
> 
> right before before "pass out quick" rules. Everything started working
> right and it worked fine for some time but then firewall died (machine
> do not respond to keyboard and the only bring it back to live was to
> bounce power). After rebooting situation repeated - router was working
> well for 2 to 10 minutes and then unexpectedly freeze. We discovered
> that problem was caused by broadcast UDP packet coming on dmz_if and
> destined to 255.255.255.255 Later we discovered that this situation is
> triggered by any packet for which both conditions are true:
> 1. destination MAC is broadcast
> 2. destination IP is none of router's directly connected networks
> 
> Any such a packet kills the router. Actually, router is not completely
> dead - it sends that damn packet over and over at huge speed to the
> outer interface.
> 
> Even if I'm doing something completely wrong, router should NOT hang that way.
> 
> 
> No idea if these bugs are PF's or they are FreeBSD specific.
> 

Okay, we tested FreeBSD 5.4, FreeBSD 6.0 and OpenBSD 3.7

Bug #2 present everywhere. Which means it is pf bug (or misfeature)

Bug #3 is definitely FreeBSD specific - it does not hurt OpenBSD.

Bug #1 does not seem like a bug anymore. In general, router can not
send reply to the interface from which packet has arrived because it
is unclear what to use as next hop on thin interface. That means this
issue is somewhat global to most flavors of UNIX. Have no idea how
this can be fixed at all - the only thing I can think about is to make
pf-generated replies processable by pf itself to be able to route them
by pf's means.

So, the final resume: FreeBSD-pf bundle has critical bug (ok, critical
for us) which we reported to FreeBSD team.