MFC of socket/protocol reference improvements

[Robert Watson]

Dear All,

I'm in the process of evaluating a possible MFC over the socket/protocol 
reference changes I made in April, 2006, to the RELENG_6 branch.  Over the 
past few months, these changes have been gradually refined, and a number of 
bugs (of varying severity) have been fixed.  These changes are important 
because they close a significant number of races, reduce the locking overhead 
and improve parallelism, and lay the groundwork for future improvement in the 
socket and protocol code.  However, they also make significant changes in a 
number of important network protocols (such as TCP), and change the semantics 
of the socket/protocol interface (protosw.h).  I think these changes are 
important to our short and long term goals of improving network stack 
performance and architecture.

My original plan was to start looking in detail at the MFC after about three 
months of settling time, which is in about three weeks.  I continue to plan to 
do this.  A few specific points for discussion:

(1) Normally, RELENG_* has significant constraints on changes to the kernel
     APIs used by loadable modules -- especially for device drivers.  In the
     past, we've not made a lot of changes to the protocol switch interface,
     and historically it hasn't been a run-time extensible interface.  Andre
     has recent made changes to allow IP protocols to be loaded at runtime,
     such as IP divert, and these will be affected, however.  Do we consider
     modules programmed against these interfaces to be "breakable" -- i.e., the
     require a recompile and or changes in the RELENG_6 branch?

(2) More testing would really be appreciated.  I caught a subtle bug in the
     handling of the retransmit timer in the context of my changes by accident
     as a result of close analysis of TCP traces through a firewall -- I
     noticed some "odd" packets that were, with a bit of time, tracked to their
     source.  However, this sort of thing is really subtle.  Any help
     determining whether there are other regressions in TCP behavior would be
     greatly appreciated.  While we've now hammered on the new code quite a
     bit, and it fixes some known panics in RELENG_6, I would categorize these
     changes as high risk, as they touch quite sensitive and heavily deployed
     code.  Getting the 7.x code tested in diverse high load environments
     before the MFC would be very good.

I'm still in the process of looking at further refinements of the 
socket/protocol relationship, which may be candidates for future merging, and 
will depend on these changes.  Among other things, I've been looking at 
further evolving the notion of socket close vs. socket detach, which are 
currently conflated notions, leading to both a lack of clarity and lack of 
flexibility in the current API.  In turn, that has presented a problem with 
experimenting with alternative locking strategies, such as vertical 
integration of locks between the socket and protocol layers.  Getting these 
changes into RELENG_6 will depend on these earlier changes being merged.

Robert N M Watson
Computer Laboratory
Universty of Cambridge