dummynet/ipfw high load?

[Dennis Yusupoff]

Good day, gurus!

We have a servers on the FreeBSD. They do NAT, shaping and traffic
accounting for our home (mainly) customers.
NAT realized with pf nat, shaping with ipfw dummynet and traffic
accounting with ng_netflow via ipfw ng_tee.
The problem is performance on (relatively) high traffic.
On Xeon E3-1270, whereas use Intel 10Gbit/sec 82599-based NIC(ix) or
Intel I350 (82579) in lagg transit traffic in 800 Mbit/sec and 100 kpps
[to customers] cause CPU load almost at 100% by interrupts from NIC or,
in case of net.isr.dispatch=deferred and net.inet.ip.fastforwarding=0.
Deleting ipfw pipe decrease load at ~30% per cpu.
Deleting ipfw ng_tee (to ng_netflow) decrease load at 15% per cpu.
Turning off ipfw (sysctl net.inet.ip.fw.enable=0) decrease load more, so
what server can pass (nat'ed!) traffic on 1600 Mbit/sec and 200 kpps
with only load ~40% per cpu.

So my questions are:
    1. Are there any way to decrease system load caused by dummynet/ipfw?
    2. Why dummynet/ipfw increase *interrupts* load, not kernel or
something like that?
    3. Are there any way to profiling that kind of load? Existing DTrace
and pmcstat examples almost useless or I just doesn't know how to do it
properly.

Huge size of debugging info (including dtrace and pmcstat samples),
sysctl settings and so on, I opened appropriate topic at russian network
operator's forum: http://forum.nag.ru/forum/index.php?showtopic=93674
In english it's available via google translate:
http://translate.google.com/translate?hl=en&sl=auto&tl=en&u=http%3A%2F%2Fforum.nag.ru%2Fforum%2Findex.php%3Fshowtopic%3D93674

Feel free to ask me any question and do actions on the server!

I would be VERY appreciate for any help and can take any measuring and
debugging on the one server. Moreover, I'm ready to give root access to
any of the appropriate person (as I already did it to Gleb Smirnoff when
we were investigate pf state problem).


-- 
Best regards,
Dennis Yusupoff,
network engineer of
Smart-Telecom ISP
Russia, Saint-Petersburg