MPD5 + DUMMYNET + PF HIGH CPU USAGE

[Steve Polyack]

  On 09/08/10 13:38, Marcos Vinícius Buzo wrote:
> Hi all.
>
> I just started working in a small WISP in a place of a friend that
> unfortunatelly is not between us anymore :(
> _ We're running FreeBSD 8.1 64bits with MPD5 for pppoe, IPFW+Dummynet for
> Traffic Shaping and PF for NAT and firewall.
> _ Our hardware is a Dell PowerEdge R210, X3430 Intel Xeon, 4GB 1066Mhz and a
> two ports Broadcom NetXtreme II BCM5716.
> _ Our WAN Link is 60mbps down/up.
>
> When we have 450+ pppoe connections and link usage is about 30mbps, things
> get strange.
> CPU usage goes to 80%+(Im using cacti+snmp to see this); we have high
> latency pings, sometimes it goes to 300ms+ and sometimes mpd5 stops doing
> its service.
>
> I did setup another server to work together, it solves the problem just for
> now, in this server i disabled flowtable (sysctl
> net.inet.flowtable.enable=0), because in the old server, when i run top
> -ISH, I see the following:
>
>   22 root      44    -     0K    16K CPU2    2 236:19 100.00% flowcleaner
>
> Is this a bug ?
>
> Are the following customizations right ?
>
> Here are the custom kernel flags:
> ...
> kern.maxvnodes=100000000
> ...

100 million vnodes sounds like a lot for a system that is not doing IO 
with lots of files.  I guess the worst it's going to do is sucking up 
some extra memory.  I can't speak much for the flowtable, but with 450+ 
clients, you are surely hitting the limits of the default number of 
entries there.

$ sysctl net.inet.ip.output_flowtable_size
net.inet.ip.output_flowtable_size: 32768
$ sysctl -d net.inet.ip.output_flowtable_size
net.inet.ip.output_flowtable_size: number of entries in the per-cpu 
output flow caches

With 4 CPUs, that tracks a maximum of 128k flows.  With 450 clients 
behind, I could see you easily exceeding that rapidly.  You may want to 
try doubling (or tripling) this value via loader.conf on the main system 
and seeing if that helps a lot (the flowcleaner may not have to 
constantly work if you are not always close to the maximum number of 
flows).  I'm not sure of the specifics of the flow table, so someone 
else could probably chime in with some more information on it (I can't 
find any real documentation on the feature).  With such a high number of 
flows, you may just be better turning it off anyways.