TCP loopback socket fusing

[Bjoern A. Zeeb]

On Mon, 13 Sep 2010, Andre Oppermann wrote:

Hey,

> When a TCP connection via loopback back to localhost is made the whole
> send, segmentation and receive path (with larger packets though) is still
> executed.  This has some considerable overhead.
>
> To short-circuit the send and receive sockets on localhost TCP connections
> I've made a proof-of-concept patch that directly places the data in the
> other side's socket buffer without doing any packetization and other protocol
> overhead (like UNIX domain sockets).  The connections setup (SYN, SYN-ACK,
> ACK) and shutdown are still handled by normal TCP segments via loopback so
> that firewalling stills works.  The actual payload data during the session
> won't be seen and the sequence numbers don't move other than for SYN and FIN.
> The sequence are remain valid though.  Obviously tcpdump won't see any data
> transfers either if the connection has fused sockets.
>
> Preliminary testing (with WITNESS and INVARIANTS enabled) has shown stable
> operation and a rough doubling of the throughput on loopback connections.
> I've tested most socket teardown cases and it behaves fine.  I'm not entirely
> sure I've got all possible path's but the way it is integrated should 
> properly
> defuse the sockets in all situations.

Three comments in reverse order:

1 If S/S+A/A and shutdown aren't shortcut, can you always rely on proper
   payload order, especially in the shutdown case?

2 Given my experience with epairs, which are basically a loop with two
   interfaces and even interface queues, any significant delay you are
   seeing is _not_ due to longer code paths through the stack but
   simply because of the netisr.

3 If properly doing this for TCP, we should probably also do it for
   other protocols.

/bz

-- 
Bjoern A. Zeeb                              Welcome a new stage of life.