pf performance issues ...

[Grooms, Matthew]

     I am seeing some pretty severe performance issues with pf+pfsync on FreeBSD 5.4-REMEASE and would like to get some advice on tuning for a largish environment. I have had some traffic moving across these firewalls for a few weeks without issue but had not pointed our default route to it until this morning.
 
     Although processor utilitzation was very low ( 2-5% ), throughput on the firewall was very very poor. TCP connections were in some cases taking 15-30 seconds to setup and in other cases never did. We had to revert our default route to an older firewall to keep operations going.
 
     This is a dual 3GHz amd64 box ( UP kernel at the moment ), with 4 gigs of ram and 6x em interfaces. It is mostly a stock kernel with pf,pfsync,carp and altq ( but no altq rules ) support compiled in and ipv6 disabled ( config attached ).
 
     Am I running into a limit on some kernel tunable? After a few minutes of routing traffic to pf setup, the state table had approx 10000 entries in it. Are there some global pf limits to tweak or should it scale well out of the box? The internet connection is only 7Mbit so I am at a loss. Is there a cache or buffer limit somewhere I should watch? Any ideas?
 
Thanks in advance,
 
Matthew Grooms