Flowtables -- any tuning hints?

[Barney Cordoba]

--- On Sun, 7/12/09, Chris Buechler <cmb at pfsense.org> wrote:

> From: Chris Buechler <cmb at pfsense.org>
> Subject: Re: Flowtables -- any tuning hints?
> To: freebsd-current at freebsd.org
> Date: Sunday, July 12, 2009, 6:41 PM
> Kip Macy wrote:
> > On Sat, Jul 11, 2009 at 10:24 AM, Scott Ullrich<sullrich at gmail.com>
> wrote:
> >   
> >> Hello Freebsd-current@ folks,
> >> 
> >> I see with the commit "svn commit: r191259 -
> head/sys/netinet"
> >> flowtables have been added..  Cool!
> >> 
> >> Does anyone have any tuning hints for this
> addition -- specifically
> >> how much memory does the hash table
> consume?   Or better yet does any
> >> documentation exist for this newly added feature?
> >> 
> >> Looking for an easy way to calculate max flows for
> the amount of
> >> memory installed in a FreeBSD machine.
> >>     
> > 
> > 
> > You want to avoid hash collisions. So, generally
> speaking you want the
> > hash table to be sized 2x larger than the number of
> unique connection
> > destinations.  You want the maximum number of
> flows to be as large as
> > the maximum number of unique destinations x number of
> cores. When you
> > get to the case of hundreds of thousands of unique
> destinations as in
> > the case of a small ISP doing IP forwarding, you're
> probably better
> > off disabling the flowtable. For most other workloads
> its likely to be
> > a clear win. Running a process on an 8-core system
> with 8 threads each
> > calling sendto(...) with 10 bytes I can push 3.5 -
> 4Mpps (with cxgb -
> > you won't get this with most cards) with the flowtable
> enabled. With
> > the flowtable disabled lock contention causes
> performance to degrade
> > to 330kpps with the aforementioned workload.
> >   
> This is interesting functionality, but I think we need to
> look at it a bit closer for our use case. Is there any
> benefit in running this in a firewall scenario? That's
> primarily what Scott and I (pfsense) are interested in. In
> our world, if you're pushing 50Kpps+, you're almost
> certainly falling into the "small ISP doing IP forwarding"
> scenario with hundreds of thousands of unique destinations.
> Where we usually see these kinds of loads are small ISPs,
> web hosting companies, or universities (which are
> functionally not much diff from a small ISP), all of which
> I'm familiar with falling into the "better off disabling"
> category. I also suspect pf's locking negates some or all of
> the benefits here.
> 
> I suspect it's not applicable to the specific workload our
> users normally have, where you're almost entirely doing IP
> forwarding, and initiating very little if any traffic. bz@
> said it's not something you want on a router. Is that a fair
> assessment? 
> best,
> Chris
> 

I don't see the need for flowtables or any sort of hashing with
multi queue boards in a pass-through environment (such as a firewall). 
The flows are already hashed by the hardware so doing all of the 
gobbledygook in flowtables seems like a waste of cycles. Just matching
your receive and transmit queues in a properly designed driver is all
thats necessary for lock avoidance.

Barney