a proposed callout API

[Matthew Dillon]

:
:     http://www.ece.rice.edu/~willmann/pubs/paranet_tr06-872.pdf
:
:Robert N M Watson

   Oh, that paper.  You know, I talked to Alan about that paper a while
   back, but it isn't really possible to compare DragonFly side by side
   with FreeBSD yet in an SMP environment because we still have a lot 
   of BGL junk in the network path, and because our interrupts are
   still going to cpu #0.  The code itself is mostly MP safe, and Jeff
   has actually turned off the BGL in some of his own testing, but I
   can't do it officially yet.  In anycase, that is why DragonFly wasn't
   used.

   I like the paper but I'm not sure it is possible to compare particulars
   of the network implementation in such different operating systems
   (FreeBSD vs Linux).

   The basic issue is, in a nutshell:

   per-packet			per-connection
   ----------			-------------
   long code paths		short code paths
   long data paths		shorter data paths
   (more cache contention)	(less cache contention, work aggregation
				has a better change of fitting in the L1/L2,
				much less 'shared' data between cpus)

   lots of mutexes		fewer mutexes or no mutexes
   (eats time, potential
   contention)

   no thread switches or	more thread switches - protocol threads
   fewer thread switches	process packets, not the interface interurpt
   (remember, interrupts	(~500ns-~1us to switch threads)
   may be threaded)

   I'm sure I can think of more differences.  The per-packet model almost
   certainly has better synchronous performance and almost certainly has
   better performance when operating on a limited number of connections.
   The basic premise is that you can make better use of any available cpu
   by using a fine-grained locking model.

   The per-connection model has the potential to process data in bulk
   with a far smaller cache footprint due to the separation of work,
   but requires a large number of connections to aggregate enough work
   to absorb the thread switching overhead.  The basic premise is that
   you only care about peformance if you are actually running your machine
   flat out.

   This is largly borne out by the paper, though I again caution that
   the operating systems are so different from each other I don't think
   the graphs can be taken as an indication of the relative benefits of
   the networking model.

					-Matt
					Matthew Dillon 
					<dillon at backplane.com>