Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

[Robert Watson]

On Mon, 7 Jul 2008, Bruce Evans wrote:

> I use low-end memory, but on the machine that does 640 kpps it somehow has 
> latency almost 4 times as low as on new FreeBSD cluster machines (~42 nsec 
> instead of ~150).  perfmon (fixed for AXP and A64) and hwpmc report an 
> average of 11 k8-dc-misses per sendto() while sending via bge at 640 kpps. 
> 11 * 42 accounts for 442 nsec out of the 1562 per packet at this rate.  11 * 
> 150 = 1650 would probably make this rate unachievable despite the system 
> having 20 times as much CPU and bus.

Since you're doing fine-grained performance measurements of a code path that 
interests me a lot, could you compare the cost per-send on UDP for the 
following four cases:

(1) sendto() to a specific address and port on a socket that has been bound to
     INADDR_ANY and a specific port.

(2) sendto() on a specific address and port on a socket that has been bound to
     a specific IP address (not INADDR_ANY) and a specific port.

(3) send() on a socket that has been connect()'d to a specific IP address and
     a specific port, and bound to INADDR_ANY and a specific port.

(4) send() on a socket that has been connect()'d to a specific IP address
     and a specific port, and bound to a specific IP address (not INADDR_ANY)
     and a specific port.

The last of these should really be quite a bit faster than the first of these, 
but I'd be interested in seeing specific measurements for each if that's 
possible!

Thanks,

Robert N M Watson
Computer Laboratory
University of Cambridge