kevent and multiple worker threads

[John-Mark Gurney]

Christof Meerwald wrote this message on Sun, Mar 18, 2012 at 22:37 +0100:
> I am trying to figure out how kqueue/kevent works in a multi-threaded
> environment. So I have created a small test program
> (http://svn.cmeerw.net/src/nginetd/trunk/test/kqtest.cc) that creates
> a socketpair and send 4-byte messages between the 2 sockets. There is
> only ever 1 message in transit, but there can be multiple threads
> waiting for event notification using kevent.
> 
> Note that I have only been able to test this inside a KVM virtual
> machine with FreeBSD 9.0 on a dual-core AMD64 laptop.
> 
> But what I find extremely interesting is that the time seems to
> increase exponentially with the number of worker threads, e.g. I get
> (the first argument is the number of worker threads and the second is
> the number of iterations):
> 
> $ ./kqtest 1 100000
> 0.508513
> $ ./kqtest 2 100000
> 1.377444
> $ ./kqtest 4 100000
> 2.533485
> $ ./kqtest 8 100000
> 12.305741

I believe this is because you are not using _ONESHOT or _DISPATCH to
remove/disable the event from the queue such that only one thread will
service the event...  Then once the event has been services, you can
re-add or re-enable the event to keep getting events...

Make sure that only one thread is being woken up and run for each
even that happens...  You can use ktrace(1) to see what threads are
doing what syscalls to ensure that only one thread wakes up...

kevent is a level triggered system, so when an event comes in, a
thread will wake up, get a copy of the event, but that event is
still active, so the next thread will get it because the first
hasn't serviced it yet, and so on, but only one thread will end up
servicing the event despite all the threads waking up...

The reason clear does not work for sockets is that the data, gets
"cleared", but then the socket is rechecked if there's data available,
and since the first thread hasn't processed it yet, there's still
data...

> BTW, I have also tested in with NetBSD 6.0 BETA on the same machine
> (also inside a KVM virtual machine) and there the results look
> different:

w/o a ktrace to compare the two, not sure why there's a difference...

Also, I do not know if NetBSD has chosen the same behavior wrt _CLEAR
as FreeBSD, or if they are different.

-- 
  John-Mark Gurney				Voice: +1 415 225 5579

     "All that I will do, has been done, All that I have, has not."