Large scale NAT with PF - some weird problem

[Milan Obuch]

On Mon, 29 Jun 2015 10:26:54 +0200
Daniel Hartmeier <daniel at benzedrine.ch> wrote:

> On Sun, Jun 28, 2015 at 10:06:09AM +0200, Milan Obuch wrote:
> 
> > So, now I am at 10.2-PRERELEASE, r284884, and the issue is still
> > here. It is totally weird, just change of IP the device is being
> > natted to makes the issue disappear for this particular customer,
> > but as soon as this exact IP is used again, the issue is here again.
> 
> Do you have access to the upstream router?
> Can you check its ARP table?

No, I do not have access here, I can't get info from there directly. I
could get some info from some admin, but this would take some time, and
I do not think it could really help me...

> It could have a static ARP entry for this specific IP address, or
> there could be an address conflict for that IP address...

Well, no reason for that, some more background below.

> Can't you tell us the network, netmask and the IP address?
> Not even with the first octet redacted?

Well, I do not like to give full details in public, but partially
redacted - all public address are from one /16 block, lets call it
x.y.0.0/16. On my side, uplink interface is em0 with IP x.y.3.19/29, on
upstream router, there is x.y.3.17/29, used as default gateway for me.
On upstream router, there is statically routed network x.y.24.0/22 to
x.y.3.19, my IP. Other IPs on uplink segment are not used currently.

From this x.y.24.0/22 address block, some smaller segments are directly
connected to my box, such as public servers (DNS, www, mail...) or some
customers with dedicated public IP. For this purpose, x.y.24.0/24
address block is used, divided into smaller segments.

Next block, x.y.25.0/24, is used mainly for binat'ed IPs, in pf.conf
one will see handfull of

binat on $if_ext from 172.a.b.c to any -> x.y.25.z

statements, and the rest, x.y.26.0/23, is used as $pool_ext, assigned
dynamically to all customers. Per Ian's advice, I am currently testing
my setup with just x.y.26.0/24 being used for NAT pool.

As for question about ARP - I think there is not anythink like static
arp on upstream router. I could ping the offending address from outside
and see them arriving on uplink interface, em0, with tcpdump. No
replies are being generated, however, but I considered this as good
evidence there is nothing blocking me on upstream router.

Does this answerred your question fully or something more would be
usefull?

Regards,
Milan