upd: 7.2->8.1 & many networks trouble & flowtable

Li, Qing qing.li at bluecoat.com
Wed Nov 24 16:21:28 UTC 2010

You actually haven't answered my questions.
I think you are reporting multiple issues in your original email, which
include issues in both userland applications as well as kernel issues (that
may be related to flow-table being enabled).
The paper discussed quite a few topics, in the right sequence, but
if you jumped ahead or jumped around, you may make unintended assumptions.
One of the main reasons for the flow-table enhancement, as its name
implies, is to "affinitize" TCP/UDP flows to specific route/interface,
even when ECMP is enabled, and the route entries in the ECMP group
are changing or being shuffled constantly. This feature is specifically
important when an appliance is deployed, for example, as a reverse
The other benefit of the flow-table, is to reduce the L3 route table
and L2 ARP/ND6 lookups by caching the search result with the connection.
In earlier releases of FreeBSD there is a field called "inp_route" 
designed for this exact purpose, however, I believe it was removed
back in FreeBSD 5.3 release.
So to summarize, the flow-table work is necessary and important,
though there may be bugs that we need to fix.
Flow-table's main benefit, as it stands currrently, is mainly for
L4 connections, not for L3 forwarding purposes.
When we were doing performance analysis through L4 connections to 
measure the benefits of separating L2/L3, as noted in the paper, the 
performance gain was not at the expected level. Further analysis showed 
there were still lock contentions due to L2 table lookup. This was 
the other motivation for the flow-table work.
I have done performance evaluation at L3 for packet forwarding tests
with 100s of route entries, and I have not seen any degradation 
compared with 7.2.
Recently we ran 8.1 on i7 processor using Avalanche testbed for performance 
evaluation, and noticed the locking contention is still very high in TCP 
connection setup and tear-down. The CPU utilization is also high due to the  
lock contentions, not due to flow-table feature because it was disabled.
So before you conclude all of the issues that you are encountering falls
within flow-table, I urge you to articulate the issues with more details.
Also, once you disable flow-table through sysctl, what issues
are you still running into.
Yes, I personally consider the flow-table work still being experiemental.
More work is being done as we speak. In addition, we are considering other 
enhancements for the routing code.
-- Qing


From: Andrey Groshev [mailto:greenx at yartv.ru]
Sent: Wed 11/24/2010 4:04 AM
To: Li, Qing
Cc: freebsd-stable at freebsd.org
Subject: Re: upd: 7.2->8.1 & many networks trouble & flowtable

24.11.2010 13:18, Li, Qing ?????:
> I am the main author of this paper you referenced in your email.
Hi! I know that you also worked on this. Kip Macy mention because I
found his statement regarding this issue.
> The main discussion and focus of my paper was on the design and work done to separate L2 and L3 for both IPv4 and IPv6 to facilitate the elimination of GIANT lock in the networking subsystem, thus achieving high parallelism.
> This redesign of separately managing L2 ARP/ND6 and L3 routing tables already show performance gain on multicore systems.
> The flow-table enhancement is just one other component, described towards the end of the paper. Yes, It is experimental and was discussed as such in the paper as well as on the mailing list.
Ie You also confirms that this feature is still experimental?
> I did not know flow-table feature was enabled by default. I wouldn't have done so myself.
Kip Macy added it to the generic kernel of head 2009-06-14 (vers. 1.526).
And it so happened that when he appeared RELENG_8 she moved into the
stable branch.
> So help me understand you better: are you complaining about the general L2/L3 separation work, or you are angry about the flow-table enhancement in particular?
> cheers,
> -- Qing
I understand the importance and necessity of the features.
I'll be glad when it will actually carry out what should be.
But in the current situation, this feature should not be enabled by
default in the generic kernel of the stable branch.

Best regards,
Andrey Groshev.

More information about the freebsd-stable mailing list