Large scale NAT with PF - some weird problem

[Milan Obuch]

On Tue, 23 Jun 2015 09:49:57 +0200
Ian FREISLICH <ian.freislich at capeaugusta.com> wrote:

> Milan Obuch wrote:
> > As a first step, I did small upgrade, so now I run FreeBSD
> > 9.3-STABLE #0 r284695: Mon Jun 22 08:55:29 CEST 2015.
> > 
> > I still see the issue, but I found simpler workaround when bad state
> > ocurs - using
> > 
> > pfctl -k <ip.of.affected.client>
> > pfctl -K <ip.of.affected.client>
> > 
> > in this order seems to remedy the issue for this one affected client
> > without affecting other clients. This still does not solve the
> > problem, just eases the reaction.
> 
> How is your NAT rule defined?  I had a closer look at the way I did
> it:
> 
> nat on vlan46 from 10.8.0.0/15 to !<on-our-net> -> xx.xx.xx.xx/24
> round-robin sticky-address
> 
> I think you may be missing the "round-robin" that spreads the mapping
> over your pool.  The manual says that when more than 1 address is
> specified, round-robin is the only pool type allowed, it does not
> say that when more than 1 address is specified this is the default
> pool option.
>

Thanks for hint, however, this is not the case I think. My definition is

nat on $if_ext from <net_int> to any -> $pool_ext round-robin sticky-address

where <net_int> contains contains some /24 segments from 10.0.0.0/8
range and one /24 and one /15 segment from 172.16.0.0/12 range,
$pool_ext is one /23 public segment.

> You can check your state table to see if it is indeed round-robin.
> 
> #pfctl -s sta |grep " ("
> ...
> all tcp a.b.c.d:53802 (10.0.0.220:42808) -> 41.246.55.66:24
> ESTABLISHED:ESTABLISHED all tcp a.b.c.e:60794 (10.0.0.38:47825) ->
> 216.58.223.10:443       ESTABLISHED:FIN_WAIT_2
> 
> If all your addresses "a.b.c.X" are the same, it's not round-robin
> and that's your problem.
> 

Well, this is something I do not fully understand. If my pool were
a.b.c.0/24, then what you wrote could not be any other way - I think
this is not what you meant. Or did you think there will be only one IP
used? That's definitelly not the case, I see many IPs from my /23
segment here.

One strange thing occured, however - it looks like if one IP from
this /23 range gets used, trouble occurs. I do pfctl -k and pfctl -K
for this address and all is well again. As long as this one IP is not
used, everything works. When it gets used again, voila, trouble again.

As this does not occur that fast, I need to check every now and then,
and I am checking the other way too, but it is really annoying if it
hits any customer.

Regards,
Milan