kern/129103: [ipfw] IPFW check state does not work =(

[Ian Smith]

Kes,

your message didn't make it to the list.  Probably because of the 650KB 
attached diagram :)  Big stuff is better put up as an URL to a web site.

On Tue, 25 Nov 2008, KES wrote:
 > #ipfw -ed show
 > 00001    0       0 check-state log
 > 00002    2     120 count log icmp from any to any via ng0
 > 00003    2     120 prob 0.500000 skipto 6 log icmp from any to any via ng0
 > 00004    0       0 skipto 5 log icmp from any to any via ng0 keep-state
 > 00005    0       0 skipto 10 log icmp from any to any via ng0
 > 00006   11     660 skipto 7 log icmp from any to any via ng0 keep-state
 > 00007    6     360 count log icmp from any to any via ng0
 > 00010    6     360 count log icmp from any to any via ng0
 > 00049  726  132682 nat 1 ip from any to any via ng0
 > 00050 9088 2196349 allow ip from any to any
 > 00050    0       0 nat 1 ip from any to any via ng0
 > 00100    0       0 allow ip from any to any via lo0
 > 00200    0       0 deny ip from any to 127.0.0.0/8
 > 00300    0       0 deny ip from 127.0.0.0/8 to any
 > 65535    0       0 deny ip from any to any
 > ## Dynamic rules (2):
 > 00006    7     420 (1s) STATE icmp 192.168.9.4 0 <-> 213.180.193.123 0
 > 00006    2     120 (1s) STATE icmp 213.180.193.123 0 <-> 92.113.76.40 0

Ok, this one is easier because no packets took the 4/5 prob split.

Note 2 separate state entries because of the different address pairs.

 > Nov 24 23:54:25 home kernel: ipfw: 2 Count ICMP:8.0 192.168.9.4 213.180.193.123 out via ng0
 > Nov 24 23:54:25 home kernel: ipfw: 3 SkipTo 6 ICMP:8.0 192.168.9.4 213.180.193.123 out via ng0
 > Nov 24 23:54:25 home kernel: ipfw: 6 SkipTo 7 ICMP:8.0 192.168.9.4 213.180.193.123 out via ng0
 > Nov 24 23:54:25 home kernel: ipfw: 7 Count ICMP:8.0 192.168.9.4 213.180.193.123 out via ng0
 > Nov 24 23:54:25 home kernel: ipfw: 10 Count ICMP:8.0 192.168.9.4 213.180.193.123 out via ng0

1st packet.  At 6 it creates state 192.168.9.4 <--> 213.180.193.123

 > Nov 24 23:54:25 home kernel: ipfw: 2 Count ICMP:0.0 213.180.193.123 92.113.76.40 in via ng0
 > Nov 24 23:54:25 home kernel: ipfw: 3 SkipTo 6 ICMP:0.0 213.180.193.123 92.113.76.40 in via ng0
 > Nov 24 23:54:25 home kernel: ipfw: 6 SkipTo 7 ICMP:0.0 213.180.193.123 92.113.76.40 in via ng0
 > Nov 24 23:54:25 home kernel: ipfw: 7 Count ICMP:0.0 213.180.193.123 92.113.76.40 in via ng0
 > Nov 24 23:54:25 home kernel: ipfw: 10 Count ICMP:0.0 213.180.193.123 92.113.76.40 in via ng0

1st response packet .. doesn't match the above state as it's a different 
address, so it creates state between 213.180.193.123 <--> 92.113.76.40 
.. because you've put your nat rule in the wrong place, it should go 
before any of this, then all state would be between 213... and 92...

Note both of the above count at rule 2, because neither match existing 
state.  These two are also the last packets that do match rule 2.

 > Nov 24 23:54:25 home kernel: ipfw: 6 SkipTo 7 ICMP:0.0 213.180.193.123 192.168.9.4 out via ng1

So now, after NAT, this matches the first state above.  It still matches 
rules 'via ng0' because that was its source interface on keep-state.  It 
doesn't show in rule 2 because it jumps to rule 6 at the check-state.

 > Nov 24 23:54:26 home kernel: ipfw: 6 SkipTo 7 ICMP:8.0 192.168.9.4 213.180.193.123 in via ng1

Second ping.  Despite coming in via ng1, it also matches the first state 
flow above, but doesn't match 7 or 10 (wrong interface)

 > Nov 24 23:54:26 home kernel: ipfw: 6 SkipTo 7 ICMP:8.0 192.168.9.4 213.180.193.123 out via ng0

Another packet out, matches established state for first rule 6 ..

 > Nov 24 23:54:26 home kernel: ipfw: 7 Count ICMP:8.0 192.168.9.4 213.180.193.123 out via ng0
 > Nov 24 23:54:26 home kernel: ipfw: 10 Count ICMP:8.0 192.168.9.4 213.180.193.123 out via ng0

.. and also matches 7 and 10, being via ng0.

 > Nov 24 23:54:26 home kernel: ipfw: 6 SkipTo 7 ICMP:0.0 213.180.193.123 92.113.76.40 in via ng0
 > Nov 24 23:54:26 home kernel: ipfw: 7 Count ICMP:0.0 213.180.193.123 92.113.76.40 in via ng0
 > Nov 24 23:54:26 home kernel: ipfw: 10 Count ICMP:0.0 213.180.193.123 92.113.76.40 in via ng0

Return packet matches second state entry on rule 6, from check-state.

 > Nov 24 23:54:26 home kernel: ipfw: 6 SkipTo 7 ICMP:0.0 213.180.193.123 192.168.9.4 out via ng1

Same packet matching first state entry on rule 6, despite interface ng1, 
after NAT.

 > Nov 24 23:54:27 home kernel: ipfw: 6 SkipTo 7 ICMP:8.0 192.168.9.4 213.180.193.123 in via ng1

Third ping packet from ng1, also matches the first of rule 6 states, but 
not counted by 7 or 10 because of via different interface.

 > Nov 24 23:54:27 home kernel: ipfw: 6 SkipTo 7 ICMP:8.0 192.168.9.4 213.180.193.123 out via ng0

Same packet going out, matches second of the rule 6 states ..

 > Nov 24 23:54:27 home kernel: ipfw: 7 Count ICMP:8.0 192.168.9.4 213.180.193.123 out via ng0
 > Nov 24 23:54:27 home kernel: ipfw: 10 Count ICMP:8.0 192.168.9.4 213.180.193.123 out via ng0

.. which also gets counted by 7 and 10, being out via ng0.

 > Nov 24 23:54:27 home kernel: ipfw: 6 SkipTo 7 ICMP:0.0 213.180.193.123 92.113.76.40 in via ng0

Return packet, matches second rule 6 state ..

 > Nov 24 23:54:27 home kernel: ipfw: 7 Count ICMP:0.0 213.180.193.123 92.113.76.40 in via ng0
 > Nov 24 23:54:27 home kernel: ipfw: 10 Count ICMP:0.0 213.180.193.123 92.113.76.40 in via ng0

.. and 7 and 10.

 > Nov 24 23:54:27 home kernel: ipfw: 6 SkipTo 7 ICMP:0.0 213.180.193.123 192.168.9.4 out via ng1

Third reply being delivered, after NAT, to ng1 because it matches state.

That's it.  If you'd had separate rules to count icmp via ng1 you'd have 
seen them all (or if you'd specifed ng* rather than ng0)

=======

 > Seems all is ok, except
 > 00001    0       0 check-state log
 > Once dynamic rule is matched, check-state counter must be increased.

No use saying 'must', when that's simply not how it works; check-state 
never shows any counts.  You've filed a PR on the basis of how you wish 
it might work, and I believe I've shown above that it's working as 
advertised.  I suggest you should now close this PR.

=======

 > Also it seems when keep-state create new dynamic rule, it must
 > increment it. Because of when rule body is matched, keep-state create
 > dynamic rule and then packet flow through this dynamic rule, counter
 > for dynamic rule is incremented, packet for parent rule is
 > incremented. and you do not loose counters.

Again, that's just not how it works.  Dynamic rule counts show traffic 
due to established state; they don't include the original packet that 
created the state.  Sometimes you'll see dynamic rule counts of 0, where 
a packet went out (say) and setup state, but no response came back.

In short, there's no use trying to use statistics from keep-state rules 
for accounting.  Use separate stateless rules, BEFORE any check-state.

 > But now NOTICE: 11 packets for rule 6 and 9 (7+2) packets for dinamic rule
 > with rule 6 as parent. 11 != 9, so here couter lost 2 packets =(

Plus the original 2 trigger packets, shown above at rule 2, making 11 
packets shown at rule 6.  I'm not entirely sure why the total of 12 
packets isn't shown (6 packets, each both in and out as you've not 
specified direction on your rules), but suspect it's to do with the 
position of your NAT rule, and/or the fact that you're not showing 
counts for 'via ng1'.
 
 > Another usefull feature is that that check-state must have body.
 > You must allow user to check same conditions as for other rules. For
 > example:
 > check-state icmp via ng0
 > It will prevent user from having unexpected maching for flows on other
 > interfaces  (ng1 interface as in my example).
 > Also I think it will be handy to have many tables for dynamic rules.
 > It is like to have many routing tables. For example:
 > ipfw add xxx check-state 1 via ng0
 > ipfw add xxx check-state 2 via ng1
 > ipfw add xxx skipto yyy icmp via ng0 keep-state 1
 > ipfw add xxx skipto yyy icmp via ng1 keep-state 2

I suggest you'll need to study the code of /usr/src/sbin/ipfw/ipfw2.c 
and /sys/netinet/ip_fw*.[ch] to find out just how it works now, before 
writing code to implement the above.  I can't see anyone else taking an 
interest in implementing such a thing (but I've been wrong before :)

 > ipfw is flexible firewall so it must remain its flexibility
 > But now ipfw has not its flexibility for check-state/keep-state because of
 > it does not allow to control what I want to check-state/keep-state

You just need to understand how ipfw stateful rules really work, and 
code your rules to match that; then I expect you will enjoy success.

cheers, Ian
(professing no more than a VERY minimal grasp of this code ..)