Large scale NAT with PF - some weird problem

[Milan Obuch]

On Sat, 20 Jun 2015 17:38:05 +0200
Ian FREISLICH <ian.freislich at capeaugusta.com> wrote:

> Hi,
> 
> How many NAT states in your table?
>

How can I find out? Is there another statistics collected I can gert
out of pfctl?

Full output of pfctl -vs info is attached (hopefully it will come
unmangled through), is there anything interesting?

> I had a router translating a /20 and a /22 to a /24 and doing
> transparent interception of those and a /16 to a proxy pool and I
> never saw this. My state table was about 380000 to 850000 with a
> search rate about quadruple yours.
>

Did you do any pf tuning? What about limits in your case?

> If you can, give 10-STABLE a try. I ran the above router pair as
> 10-CURRENT for a long time.  There are some significant performance
> improvements.
> 

It is possible, but not easy (read: needs some planning before switch)
so as not interrupt the operation.

> Ian
> 

Thanks,
Milan

> On 19 June 2015 09:24:22 Milan Obuch <freebsd-pf at dino.sk> wrote:
> 
> > Hi,
> >
> > I am managing FreeBSD 9 based router for a network using PF for
> > NAT. I think I can call it large scale - there is approximately
> > 3000 customers' devices (home routers and similar) with private IPs
> > in segment 172.16.0.0/12 translated to /23 public address block.
> > Basically, in pf.conf, there is
> >
> > nat on $if_ext from $net_int to any -> $pool_ext round-robin
> > sticky-address
> >
> > and handful of
> >
> > binat on $if_ext from 172.16.x.y to any -> a.b.c.d
> >
> > statements. It works, basically, but for some time now there are
> > some intermitent outages. When it occurs, customer's device loses
> > access to internet. I can verify it with simple ping to any address
> > outside of the network.
> >
> > The weird thing is, I can see icmp request packets coming out of
> > external interface, but no icmp echo packets coming back. While I
> > can't verify on uplink router that these replies are actually
> > coming in on interface, I am pretty sure it does, but they are not
> > visible in tcpdump's output. (When I am pinging some device outside
> > of the network, which is under my control, I can see there both
> > icmp requests and icmp echo packets. Also, if I ping address to
> > which thich ping is translated from outside, I see it on external
> > interface coming in.)
> >
> > I think I have a problem with same table being too small, but no
> > idea where it is. It is not state table, I have
> >
> > set limit states 500000
> >
> > in my pf.conf, and pfctl -vs info tells
> >
> > State Table                          Total             Rate
> >   current entries                    36668
> >   searches                      1996138369        29280.5/s
> >   inserts                         15757727          231.1/s
> >   removals                        15770004          231.3/s
> >
> > so I think I have plenty of room here. It was set in past when
> > issue a bit similar occured and using bigger state table solved it.
> >
> > Also, pfctl -vs state | grep <ip.address.with.problem> shows states
> > for not working ping as
> >
> > all icmp a.b.c.d:538 <- 172.16.x.y:538       0:0
> > all icmp e.f.g.h:40011 (172.16.x.y:538) -> a.b.c.d:40011       0:0
> >
> > where a.b.c.d is address being used as ping target (outside of
> > network), 172.16.x.y is address of device with trouble access to
> > internet, and e.f.g.h is translated address for this device,
> > allocated dynamically.
> >
> > After doing /etc/rc.d/pf restart if works again, so I think, again,
> > issue is with some table being too small. Restart empties it and
> > things begin to work.
> >
> > Does this sound familiar to anybody? I was trying to find some
> > tuning guide for pf and large scale nat, but no success yet. I
> > would be gratefull for any help.
> >
> > Regards,
> > Milan
> >
> 
-------------- next part --------------
Status: Enabled for 1 days 08:40:03           Debug: Urgent

Hostid:   0xdc96c05a
Checksum: 0xeb207be7b007c11da3f9be3c541d90d0

State Table                          Total             Rate
  current entries                    49518               
  searches                      4203116524        35739.9/s
  inserts                         31009556          263.7/s
  removals                        31000311          263.6/s
Source Tracking Table
  current entries                      548               
  searches                         8672696           73.7/s
  inserts                            11348            0.1/s
  removals                           10800            0.1/s
Counters
  match                           98746186          839.7/s
  bad-offset                             0            0.0/s
  fragment                             208            0.0/s
  short                                341            0.0/s
  normalize                              0            0.0/s
  memory                                 0            0.0/s
  bad-timestamp                          0            0.0/s
  congestion                             0            0.0/s
  ip-option                              0            0.0/s
  proto-cksum                            0            0.0/s
  state-mismatch                     44311            0.4/s
  state-insert                          12            0.0/s
  state-limit                            0            0.0/s
  src-limit                             66            0.0/s
  synproxy                               0            0.0/s
Limit Counters
  max states per rule                    0            0.0/s
  max-src-states                         0            0.0/s
  max-src-nodes                          0            0.0/s
  max-src-conn                         377            0.0/s
  max-src-conn-rate                    276            0.0/s
  overload table insertion             649            0.0/s
  overload flush states                649            0.0/s