[Bug 229092] [pf] [pfsync] States created by route-to rules pfsynced without interface

[bugzilla-noreply at freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229092

--- Comment #16 from Kajetan Staszkiewicz <vegeta at tuxpowered.net> ---
(In reply to Kristof Provost from comment #15)

> (In reply to Kajetan Staszkiewicz from comment #13)
>
>> - Any rule using interface IP addresses in unnamed table {} will end up 
>> being different on 2 routers unless named <table> {} is used.
>
> Ah, because pf generates a random id for the table?

I think so.

> I'd argue that that's
> something the rules sync script (if there is one)

I don't "sync" rules. I "generate" on central database and upload to
loadbalancers. Generated files look identical, line by line. (+/- Python
issue, I will comment on it later).

> should account for, but I'd

Taking that into account is exactly what was needed in my case. Consider
such two rules:

1. allow in on $IFACE from { $HOST1 $HOST2 }

Table used here is unnamed, anonymous, dynamic or however it is called
in the world of pf. There is no guarantee of its name and thus even if
configuration is generated centrally, it will result in ruleset having
different checksum on each loadbalancer.

But is there even any real table used at all? I remember something about
dynamically generated table names but what I see is expansion of ruleset
during loading into separate rules. e.g. rule:

rdr on $if_public inet6 proto ipv6-icmp from any to $if_public ->
<own_addrs_pub>

got expanded to 2 rules:

rdr on public inet6 proto ipv6-icmp from any tofe80::6a05:caff:fe0b:dd02 ->
<own_addrs_pub> round-robin
rdr on public inet6 proto ipv6-icmp from any to 2a00:XXXXXX -> <own_addrs_pub>
round-robin

(BTW, expansion to link-local addresses seems a bug to me, I will report
it separately).

2. table <table> { $HOST1 $HOST2 }
pass in on $IFACE from <table>

Here table is named. Ruleset is now consitent between loadbalancers no
matter the contents of table.

> be happy to take patches to make that 'random id' predictable (and consistent
> across hosts).

Maybe one day but for now I already forced usage of named tables everywhere.

>> - Same thing for SNAT rules, although I'm unsure if those are included in pfchecksum.
>
> I'm not sure what you mean by SNAT rules.

Sorry, of course I meant NAT rules in pf. I very much prefer nftables
terminology of SNAT and DNAT, they just make way more sense.

> The pf_setup_pfsync_matching()
> function checksums all rules, other than the scrub rules.

That just adds one more type of rules that can screw up checksum, as I
expected.

>> - If ruleset is dynamically generated by a script, data structure might not 
>> have explicit ordering and produce different result on each run: for me it
>> was Python and its dictionaries and sets.
>
> I don't understand this one. 

Data structures like sets and hashes have no explicit ordering, at least
in Python. I think I was getting consistent results with Python 2.7 but
totally random when moved to 3.5 Things put to them will be retrieved in
some random order. One database of rules will produce functionally
identical (at least as long as they are "quick" rules) firewall but with
rules in different order. Of course pf can't do anything about it and
this is expeced, see next paragraph.

> It shouldn't matter how rules are generated, the
> kernel will calculate a checksum. Or do you mean to say pf should compensate
> for bugs 

That is not a bug.

> in synchronisation scripts?

No, it definitely should not. All I'm saying that it is another trap
I've encountered while fighting with this topic and that it is very hard
to make the ruleset identical from point of view of pf and we should not
expect identical rulesets.

> I don't really see a way around the requirement for the ruleset to be identical
> on all pfsync synced hosts.

But is there really such requirement with current status of pf?

I think the whole discussion wandered away from the main topic. Let's
get back on track.

Current situation:

1. Identical pf.conf will result in different checksum in many cases due
to interface addresses, dynamic table names and/or rule expansion from
unnamed tables.
2. pfsync of normal firewall states which only pass or nat traffic don't
need identical ruleset at all.
3. pfsync of states from route/dup/reply-to rules is *fully broken*.

Let me repeat once again: none of *working* functionalities of pf seems
to require identical ruleset. Maaaaybeee label counters?

I want to focus on fixing issue 3. There are multiple aproaches:
1. Old patch which depends on ruleset being identical and reconstructing
missing information from rules.
2. New patch which sends part of missing information (state->rt) over
pfsync and discovers interface to use from normal route lookup.
3. Modify pfsync structure to io include both state->rt and state->rt_kif.

I would *love* to have 3. implemmented but for now I work with 2.
because 1. was way too unrealiable.

How should we progress?

-- 
You are receiving this mail because:
You are the assignee for the bug.