SOCK_DGRAM optimization needed...

[Robert Watson]

On Fri, 18 Aug 2006, Martin Eugen wrote:

> I have a simple application, that deals with lots of dgram sockets (UDP). 
> Thousands of them. Basically, its purpose is to maintain pairs of sockets 
> and when data is received on one of the sockets it peeks through it (doing 
> some simple statistic calculations) and then forwards it to the other 
> socket. Because of the hudge number of reads and writes (probably about a 10 
> packets per second per socket pair) it generates a significant load on the 
> system, that I would like to minimize. I'm currently evaluating if it would 
> be possible to add simple 'routing' functionality in the socket layer in the 
> kernel, because frankly I'm not able to think of anything else.

There are some SOCK_DGRAM optimizations present in 7.x for fast UDP send not 
currently present in 6.x that may be relevant, as they reduce the overhead 
associated with socket send on a pure datagram socket, as well as contention 
if simultaneous sends occur on the same socket.  How to manage events and 
context switches is presumably also critical -- using kqueue() instead of 
poll() or select() may make a big difference, and you want to avoid force 
context switches per packet, so using an event based model rather than a 
threaded model makes a great deal of sense (if it meets other security and 
architectural requirements), as for small amounts of calculation, context 
switch cost will out-weight the benefits of concurrency unless concurrency is 
designed into the system very carefully.

However, if performance is really a critical issue here, it sounds like you 
might want to think about pushing this into the kernel.

When you say SOCK_DGRAM, do you mean UNIX domain sockets or UDP sockets?  (Or 
something else, for that matter)?

Robert N M Watson
Computer Laboratory
University of Cambridge