how much memory does increasing max rules for IPFW take up?

[Ian Smith]

On Thu, 15 May 2008, Jeremy Chadwick wrote:
 > On Thu, May 15, 2008 at 11:03:53AM +0100, Bruce M. Simpson wrote:
 > > Andrey V. Elsukov wrote:
 > >> Vivek Khera wrote:
 > >>> I had a box run out of dynamic state space yesterday.  I found I can 
 > >>> increase the number of dynamic rules by increasing the sysctl parameter 
 > >>> net.inet.ip.fw.dyn_max.  I can't find, however, how this affects memory 
 > >>> usage on the system.  Is it dyanamically allocated and de-allocated, or 
 > >>> is it a static memory buffer?
 > >>
 > >> Each dynamic rule allocated dynamically. Be careful, too many dynamic 
 > >> rules will work very slow.
 > >
 > > Got any figures for this? I took a quick glance and it looks like it just 
 > > uses a hash over dst/src/dport/sport. If there are a lot of raw IP or ICMP 
 > > flows then that's going to result in hash collisions.
 > >
 > > It might be a good project for someone to optimize if it isn't scaling for 
 > > folk. "Bloomier" filters are probably worth a look -- bloom filters are a 
 > > class of probabilistic hash which may return a false positive, "bloomier" 
 > > filters are a refinement which tries to limit the false positives.
 > >
 > > Having said that the default tunable of 256 state entries is probably quite 
 > > low for use cases other than "home/small office NAT gateway".
 > 
 > It's far too low for home/small office.  Standard Linux NAT routers,
 > such as the Linksys WRT54G/GL, come with a default state table count of
 > 2048, and often is increased by third-party firmwares to 8192 based on
 > justified necessity.  Search for "conntrack" below:
 > 
 > http://www.polarcloud.com/firmware
 > 
 > 256 can easily be exhausted by more than one user loading multiple HTTP
 > 1.0 web pages at one time (such is the case with many users now have
 > browsers that load 7-8 web pages into separate tabs during startup).
 > 
 > And if that's not enough reason, consider torrents, which is quite often
 > what results in a home or office router exhausting its state table.
 > 
 > Bottom line: the 256 default is too low.  It needs to be increased to at
 > least 2048.

I think there may be some confusion in terms.  Looking at defaults on my
older 5.5 system - sure, call it a "home/small office NAT gateway": 

 net.inet.ip.fw.dyn_buckets: 256
 net.inet.ip.fw.curr_dyn_buckets: 256
 net.inet.ip.fw.dyn_count: 212
 net.inet.ip.fw.dyn_max: 4096
 net.inet.ip.fw.static_count: 153

What defaults to 256 is the number of hash table buckets, not the max
number of dynamic rules, here 4096 (though the 5.5 manual says 8192).

On hash collisions, a linked list is used for duplicate hashes of:

        i = (id->dst_ip) ^ (id->src_ip) ^ (id->dst_port) ^ (id->src_port);
        i &= (curr_dyn_buckets - 1);

So while 256 may well be too few buckets for many systems, and like
Bruce I wonder about the effectiveness of the xor hash for raw IP & ICMP
and wouldn't mind seeing some stats on bucket use vs linked list lengths
for various workloads, it doesn't determine the max no. of dynamic rules
available, which is adjustable up without any apparent static memory
allocation, and is moderated by the various expiry timeout sysctls. 

For reference, I admin a 4.8 filtering bridge with up to 20 boxes behind
it, that has only very rarely reported exceeding the max no. of dynamic
rules with the (4.8) default net.inet.ip.fw.dyn_max of 1000 .. however
it only keeps state for UDP connections (and yes, it only ever hits that
limit on torrents or skype, which are generally admin. prohib. :)

cheers, Ian  (not subscribed to -ipfw)