ng_netflow and router performance question

[Ivan Alexandrovich]

Hi

I'd like to ask a question about performance tuning for freebsd
router running ng_netflow. There were a number of messages in
mailing lists and forums where people inform that in their setups
ng_netflow sustains high packet rates (>100K pkt/s) without any
harm to accounting accuracy and lost packets. Nobody mentions
any specific tuning so it can be assumed that in many cases
it just runs 'out of the box' with default kernel parameters
set (sysctls, boot-time and compile-time).

So I tried ng_netflow and got very strange result: with 13K pkt/s
and 25K different flows the router looses most of the packets.
I know there's something wrong with my setup, and not with netgraph
code itself but I just cant't figure out why it is so.
When packet do not pass ng_netgraph router handles 60K pkt/s just fine.
But with ng_netgraph performance degrades significantly - no matter
what kernel and set of ip stack optimization options were used.

During testing I used RELENG_6 (200610) and 6.1-RELEASE.
Kernels: built for K8 architecture and GENERIC from the distribution.
Custom kernel configurations were based on GENERIC for amd64 and i386.
Just got rid of unused device drivers. Added DEVICE_POLLING,
also tried with "maxusers 512".

Linux box with pktgen was used to generate the packet flow:
ip packet size: 250; protocol: udp, src/dst port=9;
ip-addr varies from 10.60.0.0 to 10.60.100.255

Hardware configuration:
Processor: Athlon 64 x2 3800+
Motherboard: Asus K8Ne
Mem: 2x1Gb DDR-400
Hdd: Western Digital WD1600JS (SATA)
Network inetface 1: integrated nforce4 Ethernet adapter
Network interace 2: 3Com 905B pci33
1 as incoming, 2 as outgoing or vise versa
tried to use xl0 in polling mode with "options HZ=1000" in
kernel configuration.

Linux router with fprobe-ulog (userespace accounting via ipt_ULOG)
handled packet rate 43pkt/s with 256K unique flows on the same
box and 20pkt/s with  2560K unique flows. ng_netflow running
in kernel context surely can perform faster.

#route for 10.60.0.0 for testing purposes
  ifconfig nve0 10.1.1.2 netmask 255.255.255.0
  ifconfig xl0 10.50.1.1 netmask 255.255.255.0
  arp -s 10.50.1.2 3:2:1:5:6:3
  route add -net 10.60.0.0 -netmask 255.255.0.0 10.50.1.2
  sysctl -w net.inet.ip.forwarding=1
  route add default 10.1.1.1

#ng_netflow usage scenario 1
  /sbin/kldload ng_ether
  /sbin/kldload ng_ipfw
  /usr/sbin/ngctl mkpeer ipfw: netflow 10 iface0
  /usr/sbin/ngctl name ipfw:10 countall
  /usr/sbin/ngctl msg countall: setdlt { iface=0 dlt=12 }
  /sbin/ipfw add 64990 ngtee 10 ip from any to 10.60.0.0/16 in
  /sbin/ipfw add 64999 allow ip from any to 10.60.0.0/16 out
  /sbin/ipfw add 65000 allow ip from any to any
  /usr/sbin/ngctl msg countall: settimeouts { inactive=30 active=3
  /usr/sbin/ngctl mkpeer countall: ksocket export inet/dgram/udp
  /usr/sbin/ngctl msg countall:export connect inet/127.0.0.1:4444
  /usr/local/bin/flow-capture -z 0 \
   -n 600 -w /var/log/flows 127.0.0.1/127.0.0.1/4444

#ng_netflow usage scenario 2 (old-style)
  /usr/sbin/ngctl  mkpeer xl0: tee lower right
  /usr/sbin/ngctl  connect xl0: xl0:lower upper left
  /usr/sbin/ngctl  name xl0:lower xl0_tee
  /usr/sbin/ngctl  mkpeer  xl0:lower netflow  right2left iface0
  /usr/sbin/ngctl  name xl0:lower.right2left netflow
  /usr/sbin/ngctl  msg netflow: setifindex { iface=0 index=1 }
  /usr/sbin/ngctl  connect xl0:lower netflow: left2right iface1
  /usr/sbin/ngctl  msg netflow: setifindex { iface=1 index=1 }
  /usr/sbin/ngctl  mkpeer netflow: ksocket export inet/dgram/udp
  /usr/sbin/ngctl  msg netflow:export connect inet/127.0.0.1:4444
  /usr/local/bin/flow-capture -z 0 \
  -n 600 -w /var/log/flows 127.0.0.1/127.0.0.1/4444

# kernel runtime parameters that I tried to change include:
  sysctl -w net.inet.ip.fastforwarding=1
  kern.polling.enable=1
  kern.ipc.nmbcluster=32768
  kern.ipc.maxsockbufs=2097152
  kern.ipc.somaxconn=8192
  net.inet.udp.recvspace=65535
  net.inet.udp.maxdgram=57344
  net.local.stream.recvspace=65535
  net.local.stream.sendspace=65535
  net.graph.maxalloc=2048 (via loader.conf)
  net.graph.maxdgram=45000
  net.graph.recvspace=45000
  sysctl kern.random.sys.harvest.ethernet=0
  sysctl kern.random.sys.harvest.interrupt=0

According to counters of ipfw rules only small percent of packets
get through < 100000 of 50000000. With small number of unique
flows (~1000) acounting works almost without losses (99,6%).
The whole situation is just ridiculous.
Maybe someone can give a hint - what am I doing wrong?

Thanks,
Ivan