tcp connection splitter

Daniel Taylor daniel_h_taylor at yahoo.co.uk
Thu Apr 12 19:35:32 UTC 2007


Not a FreeBSD question per se but I thought I'd ask
anyway, as there are many knowledgeable people here.

I am writing a server (which btw is going to run on
FreeBSD), acting as a streaming gateway for multiple
TCP clients.

The server connects to a source of data (which
only accepts a single connection) and needs to
distribute this data to multiple TCP clients.
With the exception of the initial login sequence,
communication is mostly one-way (data source to
server, and server to clients).

What is the most efficient data structure for this
scenario?   I need something that would scale to
many client connections.

The simplest solution is to maintain a separate
packet queue for each client, and copy each packet
received from the source to all N client queues.
Of course, this solution involves a lot of memory
allocations (the server receives 1-10MB of
data/second), a lot of memcpy()s, and doesn't scale
very well.   Also, adding a packet to N queues is
expensive because it needs to acquire and release
N mutex locks (one for each client queue.)

At the moment, I have a solution which allocates a
reference-counted packet, and adds a *pointer* to it
to each of the N client queues.   This saves on
memcpy because copying a pointer is very fast.  Each
enqueue bumps the refcount, each dequeue decreases it;
when the refcount drops to 0, the packet is free()'d
(by whomever happened to dequeue it last).

If the clients are "slow" (can't keep up with the
source), it's ok to drop them; hence I'm toying with
the idea of having a single circular queue accessed
by N clients.   It would certainly save on memory
allocation, but at the same time increase lock
contention, because the server and N clients will
all be locking the same queue mutex to access it.

I could use read-write locks here (single writer,
multiple readers) but it doesn't help with write
lock contention.

There must be some clever lockless algorithm to
achieve the same effect.

Any suggestions?

Thanks,

Dan



		
___________________________________________________________ 
To help you stay safe and secure online, we've developed the all new Yahoo! Security Centre. http://uk.security.yahoo.com


More information about the freebsd-hackers mailing list