Large scale NAT with PF - some weird problem

[Milan Obuch]

On Tue, 23 Jun 2015 10:57:44 +0200
Ian FREISLICH <ian.freislich at capeaugusta.com> wrote:

> Milan Obuch wrote:
> > > all tcp a.b.c.d:53802 (10.0.0.220:42808) -> 41.246.55.66:24
> > > ESTABLISHED:ESTABLISHED all tcp a.b.c.e:60794 (10.0.0.38:47825) ->
> > > 216.58.223.10:443       ESTABLISHED:FIN_WAIT_2
> > > 
> > > If all your addresses "a.b.c.X" are the same, it's not round-robin
> > > and that's your problem.
> > > 
> > 
> > Well, this is something I do not fully understand. If my pool were
> > a.b.c.0/24, then what you wrote could not be any other way - I think
> > this is not what you meant. Or did you think there will be only one
> > IP used? That's definitelly not the case, I see many IPs from my /23
> > segment here.
> 
> I just wanted to check that more than 1 address was being used.
>

OK, it is. If only one IP were used for all traffic, I would run into
issues much earlier.

> So, I think that the problem is with 9-STABLE.  I hate "upgrade to
> solve your problems" answers because they may not.  I do know that
> 10 has seen a lot of work and none of that work will make it back
> into 9 because of the PF rewrite.  Maybe someone else in this group
> will chime in.
>

That's OK. I am a bit conservative on upgrades here because with
hundreds - thousands users you need a bit of stability too, but upgrade
to 10-STABLE is currently being prepared. That being written, it will
not occur today.

> I ran 10-CURRENT in production for as long 10 was CURRENT and then
> went to 10-STABLE precisely because I was having state issues
> forwarding performance issues with 9.  Gleb Smirnof did a significant
> rewrite of PF to improve SMP performance.  He had access to my
> system for debugging on a large installation.
>

Well, we'll see. I'll let you know how it goes when upgrade will be
done.

> If you're not already doing so, I'd recomend running CARP + pfsync
> so you can test updates while maintaining a known working backup.
> If you're running pfsync, I recommend you run it on a different
> interface to the one with your traffic and with a cross-over cable
> between your machines.  The pfsync packet rate caused a small amount
> packet loss on other network traffic.
> 

I did not experiment much with CARP and pfsync. I plan to use it, but
that means more hardware... and I am not the one who pays for it.
Anyway, I would try to redesign the whole thing so it will be easier to
maintain and, if necessary, troubleshooting.

Regards,
Milan