svn commit: r222748 - in head/sys: conf netinet netinet/ipfw netinet6

[John Baldwin]

On Monday, June 06, 2011 8:55:03 am Robert Watson wrote:
> Author: rwatson
> Date: Mon Jun  6 12:55:02 2011
> New Revision: 222748
> URL: http://svn.freebsd.org/changeset/base/222748
> 
> Log:
>   Implement a CPU-affine TCP and UDP connection lookup data structure,
>   struct inpcbgroup.  pcbgroups, or "connection groups", supplement the
>   existing inpcbinfo connection hash table, which when pcbgroups are
>   enabled, might now be thought of more usefully as a per-protocol
>   4-tuple reservation table.
>   
>   Connections are assigned to connection groups base on a hash of their
>   4-tuple; wildcard sockets require special handling, and are members
>   of all connection groups.  During a connection lookup, a
>   per-connection group lock is employed rather than the global pcbinfo
>   lock.  By aligning connection groups with input path processing,
>   connection groups take on an effective CPU affinity, especially when
>   aligned with RSS work placement (see a forthcoming commit for
>   details).  This eliminates cache line migration associated with
>   global, protocol-layer data structures in steady state TCP and UDP
>   processing (with the exception of protocol-layer statistics; further
>   commit to follow).
>   
>   Elements of this approach were inspired by Willman, Rixner, and Cox's
>   2006 USENIX paper, "An Evaluation of Network Stack Parallelization
>   Strategies in Modern Operating Systems".  However, there are also
>   significant differences: we maintain the inpcb lock, rather than using
>   the connection group lock for per-connection state.
>   
>   Likewise, the focus of this implementation is alignment with NIC
>   packet distribution strategies such as RSS, rather than pure software
>   strategies.  Despite that focus, software distribution is supported
>   through the parallel netisr implementation, and works well in
>   configurations where the number of hardware threads is greater than
>   the number of NIC input queues, such as in the RMI XLR threaded MIPS
>   architecture.
>   
>   Another important difference is the continued maintenance of existing
>   hash tables as "reservation tables" -- these are useful both to
>   distinguish the resource allocation aspect of protocol name management
>   and the more common-case lookup aspect.  In configurations where
>   connection tables are aligned with hardware hashes, it is desirable to
>   use the traditional lookup tables for loopback or encapsulated traffic
>   rather than take the expense of hardware hashes that are hard to
>   implement efficiently in software (such as RSS Toeplitz).
>   
>   Connection group support is enabled by compiling "options PCBGROUP"
>   into your kernel configuration; for the time being, this is an
>   experimental feature, and hence is not enabled by default.
>   
>   Subject to the limited MFCability of change dependencies in inpcb,
>   and its change to the inpcbinfo init function signature, this change
>   in principle could be merged to FreeBSD 8.x.
>   
>   Reviewed by:    bz
>   Sponsored by:   Juniper Networks, Inc.

FYI, this commit broke the net-snmp build in ports.  The issue is that the
alignment on 'struct inpcbgroup' added a new dependency of <sys/param.h> to
<netinet/in_pcb.h> (previously <sys/types.h> sufficed) which breaks the
conftest.c that net-snmp uses to see if <netinet/in_pcb.h> is valid.  Perhaps
we should only tag inpcbgroup with an alignment requirement in the kernel and
not in userland?  It seems we should also be careful in exposing things that
depend on CACHE_LINE_SIZE to userland.

configure:31545: checking for netinet/in_pcb.h
configure:31591: cc -c -DNETSNMP_ENABLE_IPV6 -O2 -pipe -march=pentium4 -fno-stri
ct-aliasing -Ufreebsd9 -Dfreebsd9=freebsd9  -DAPPLLIB_EXP="/usr/local/lib/perl5/
5.12.4/BSDPAN" -DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe 
-fstack-protector -I/usr/local/include  -I/usr/local/lib/perl5/5.12.4/mach/CORE 
 -I/usr/include  conftest.c >&5
In file included from conftest.c:217:
/usr/include/netinet/in_pcb.h:390: error: requested alignment is not a constant
configure:31598: $? = 1
configure: failed program was:
| /* confdefs.h.  */
....

| /* end confdefs.h.  */
| 
| #if HAVE_SYS_TYPES_H
| #include <sys/types.h>
| #endif
...
| 
| #include <netinet/in_pcb.h>

-- 
John Baldwin