MFC of socket/protocol reference improvements

[Robert Watson]

On Thu, 15 Jun 2006, Doug White wrote:

> On Sun, 11 Jun 2006, Robert Watson wrote:
>
>> (1) Normally, RELENG_* has significant constraints on changes to the kernel
>>    APIs used by loadable modules -- especially for device drivers.  In the
>>    past, we've not made a lot of changes to the protocol switch interface,
>>    and historically it hasn't been a run-time extensible interface.  Andre
>>    has recent made changes to allow IP protocols to be loaded at runtime,
>>    such as IP divert, and these will be affected, however.  Do we consider
>>    modules programmed against these interfaces to be "breakable" -- i.e., 
>> the
>>    require a recompile and or changes in the RELENG_6 branch?
>
> From a policy standpoint, breaking modules in a -STABLE branch is forbidden 
> since it causes pain for 3rd party developers. Exceptions can be made for 
> changes that provide more benefit than damage caused by breaking the ABI.
>
> Usually the question revolves around, "Does anyone actually distribute 
> modules that use that interface?" :-)

So far I've identified one third party infiniband stack used in a product that 
implements the protosw API, and would need changes.  However, the scope and 
nature of the product mean that this wouldn't be a significant issue for them. 
I guess the question is whether and how many other protocol modules exist out 
there.  My leaning is to say very few, but perhaps those people simply don't 
talk to me/us.  It would be good to know if there are any other significant 
protocol stacks being distributed in binary form.

The source changes are relatively minor to update a protocol, but do need to 
be done.  I have some more changes in the works that follow on the heels of 
this change that require a slightly larger set of protocol changes -- all in 
the management of socket setup and teardown (attach, abort, close, detach). 
The larger overhaul portion of my changes is within protocols, and those 
changes largely rely on the protocol switch changes.

These changes do a couple of things:

(1) Close a number of known races and eliminate a number of known panics.  For
     example, a panic in tcp_ctloutput() has been repeatedly reported, and is
     fixed by this because so_pcb can no longer suddenly become NULL during a
     call to setsockopt().

(2) Reduce lock contention on the tcbinfo lock by avoiding calling it in the
     socket send path in a significant number of situations -- specifically,
     normal, off-the-self send and receive on TCP.

(3) Lay the groundwork for future changes to break down tcbinfo and otherwise
     optimize TCP locking.

We can do (1) through some workarounds I've been looking at in a few cases, 
and I believe specifically in the setsockopt() case.  (2) and (3) are 
basically impossible without these changes, so if we want to get them into 
RELENG_6 as opposed to just RELENG_7, merging this set of changes is 
necessary.

I hope to post an initial draft patch in a few days -- I'm currently setting 
up a Perforce branch and merging the changes to that branch, but there are 
about 65 commits involved.

Robert N M Watson
Computer Laboratory
University of Cambridge