Netgraph performance with ng_ipfw

[Luigi Rizzo]

On Sat, Jan 23, 2010 at 03:32:11AM +0600, ??????? wrote:
> Hi, 
> I have several routers under heavy load, running FreeBSD 7.2
> These routers use Netgraph to impelement traffic shaping and accouting
> (using ng_car and ng_netflow nodes). 
> Packets are passed from firewall to netgraph using the following rules
> accounting:
> netgraph 100 ip from any to any in
> shaping:
> netgraph tablearg ip from any to table(118) out
> netgraph tablearg ip from table(118) to any in
> 
> Table 118 contains users' ip addresses with tablearg referencing configured individual ng_car node.
> At peak, there are 1500-2000 entries in table and configured nodes.
> The problem is that at peak load the router loses packets. After studying the sources & doing some debugging, 
> it became clear that packets are being droped at netgraph queue, at ng_alloc_item function:
...
> When the large number of hooks is present, as in the configuration given in the beginning of this message, 
> this would cause an obvious decrease in performance - for each packet passed from ipfw to netgraph, 
> 1 to 1500-2000 iterations are needed to find matching hook. And again, it seem to be a trivial task to rewrite 
> this code to find hook by hash or even by array.

i solved exactly this problem in my recent ipfw rewrite (in HEAD), but it
is not terribly trivial, though, because you should consider memory
costs (the cookie space is large) and the need to handle reconfigurations
without blocking the forwarding for a long time.

You can use the same techniques (and possibly code) i used in ipfw,
but i suspect it will take a good 1-2 weeks of work to have a production
quality thing (I am not familiar with netgraph code and configuration
tools so i may underestimate the work).

cheers
luigi