[Bug 219815] ipfw stops working when more than two tables are used

Tue Jun 6 05:32:06 UTC 2017

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219815

            Bug ID: 219815
           Summary: ipfw stops working when more than two tables are used
           Product: Base System
           Version: 10.3-RELEASE
          Hardware: amd64
                OS: Any
            Status: New
          Severity: Affects Many People
          Priority: ---
         Component: kern
          Assignee: freebsd-bugs at FreeBSD.org
          Reporter: ecsd at transbay.net

This is not bug 165939. I do these things before doing anything with tables:

After the original fw flush:

sysctl kern.ipc.soacceptqueue=2048
ipfw table all flush { as recommended in 165939.}

The behavior is bizarre. I started working with tables, loaded addresses into
them, added rules via programs, and the rules seemed to work ... mostly. I had
a rule for table 86 to deny ip from any to any, and despite adding addresses to
the table, some of those addresses still got through. When I added a lower
numbered rule "deny ip from blah to any" that did not use tables, then the
firewall stopped the packets.

Ever After a reboot and a system crash (due to a failing disk), I can add ONE
TABLE that works, but despite adding addresses (I know connect to the machine)
to other tables, they NEVER TRIGGER. The crash did not injure system software.

I originally manually (a) loaded the tables, (b) issued the ipfw command to
effect a rule against the table. After a reboot before the crash, not only
would the table rules refuse to trigger, higher-numbered firewall rules also
refused to trigger. Causing havoc with my mail system where the tables formerly
firewalled off intense bad guy traffic.

I then surmised that perhaps if I got rid of all the table-based firewall rules
first, then loaded tables, then introduced rules to use them, that might fake
the error out, but no. Only the first table rule shows hits and the rest
pretend they see nothing at all. It also appears that whatever this is, stops
the firing of any later-numbered rules.

Some of the tables are very large, but I phased them in from small to large,
testing between each load, and it seems that as soon as a 2nd table is
introduced, things break. I am stymied because before the reboot and later disk
crash (not affecting the system main disk) the tables were working more than
not, apart from the indication of a failure starting to occur when newly added
"block all"-rule addresses were not being blocked.

Some of the trouble tickets made reference to ipfw.conf and directories where
firewall control information is stored but man and apropos know nothing about
it, nor does the online documentation. Since things were working for a while, I
thought I had smashed a limit (when I converted the tables back into "deny"
rules there were 12,800 of them.) But nothing says there is any upper limit to
the content of a table (or any sysctl thingy to affect it.) Also I never used a
table numbered higher than 120 - the system said the limit was 127, I think,
while the man page says the limit is 1024, and then I found I could set the
limit with sysctl.

I marked this as "affects many people" because if they see what I see ... then
it does.

I see a couple trouble tickets that suggest similar funky behavior, but the
contexts are different and I see nothing that says higher-numbered rules are
ignored after tables enter the picture; I see "later rules don't /load/", not
"don't fire". Also, ipfw complains when the scripts try to re-enter a subnet
already in the table, the message is

ipfw: setsockopt(IP_FW_TABLE_XADD): File exists

but I assume that is not an issue. However, I get that message even when the
rules are sorted and run through uniq to eliminate duplicates. While the tables
were working I observed that X/N did not collide with X/M for N != M.The one
hint of something going awry in the system guts is a curious message "No prev
search." that occurs sometimes after the "add" of many many addresses, after
which the add script seems to die:

I do
ipfw table N add X1
ipfw table N add X2
...
ipfw add # deny ip from table\(N\) to any

and when the "No prev search." message appears, the command to add the rule
never executes.

Has anyone else observed similar behavior? Does anyone know how to make it stop
(short of migrating to 11.0 from 10.3, if possible.) Are there any patches.
Yada etcetera.

==

As to suggestions:

* It would be nice to be able to tell ipfw to list the rules in the order they
were added to the table. As is they are listed numerically and there is no
choice about the order.

* When ipfw complains 'XADD: file exists', it would be nice to see the address
being complained of being duplicated.

* program call ''system("ipfw -q table # add address/mask");'' still complains
about XADD dups to stderr, I thought '-q' was supposed to suppress that.

* it would be /real/ cool to have an option for 'ipfw table list', to have it
output only the largest or smallest containing rules, and/or to have 'ipfw
table add' delete a larger or smaller subnet upon insertion, e.g. the table
contains

X/24
X/16

and the modified list command would only report X/16 since X/24 is entirely
subsumed by it; or vice-versa. The insertion to delete supersets or subsets,
i.e. insert X/16 if X/24 is already present, deletes X/24 and adds X/16; or
vice-versa. This is one of those things "left as an exercise for the reader", I
think.

ipfw show, shows me (excerpt):

...
01000  59589   5790205 allow ip from table(93) to any
01000  37042   6535481 allow ip from any to table(93)
01001      0         0 allow ip from table(94) to any dst-port 25
03000      0         0 deny ip from table(86) to any
03001      0         0 deny ip from table(101) to any dst-port 20-25,110,...
03001      0         0 deny ip from table(101) to any dst-port 20-25,110,...
03004      0         0 deny ip from table(103) to any dst-port 20-25,110,...
03005      0         0 deny ip from table(59) to any dst-port 25
03008      0         0 deny ip from table(102) to any dst-port 20-22,110,...
03009      0         0 deny ip from table(53) to any dst-port 25
03010      0         0 deny ip from table(80) to any dst-port 80,443
03011      0         0 deny ip from table(50) to any dst-port 25
03012      0         0 deny ip from table(58) to any dst-port 25
03013      0         0 deny ip from table(51) to any dst-port 25
03014      0         0 deny ip from table(21) to any dst-port 20,21
03015      0         0 deny ip from table(54) to any dst-port 25
03016      0         0 deny ip from table(61) to any dst-port 25
03017      0         0 deny ip from table(52) to any dst-port 25
03018      0         0 deny ip from table(20) to any dst-port 143,993
03019      0         0 deny ip from table(98) to any dst-port 20-22,110,...
03020      0         0 deny ip from table(11) to any dst-port 25
03021      0         0 deny ip from table(55) to any dst-port 25
03022      0         0 deny ip from table(10) to any dst-port 25
03023      0         0 deny ip from table(64) to any dst-port 25
05000     10       428 deny ip from 1.0.0.0/8 to any dst-port 20-25,110,...
...

Table 93 (first added) is the only table that works. Table 94 should have
registered hits and does not, and for sure addresses were connecting that
should have added hits the the 3xxx series rules; and rule 5000 ceased to
increment after the table rules were added, despite knowing for sure that more
hits should have accrued to it.

For jollies to know, table 100 refers to all of CN, and contains some 7700
entries. 103 refers to all of VN, for 550 entries.

-- 
You are receiving this mail because:
You are the assignee for the bug.