kevent and multiple worker threads

[Christof Meerwald]

Hi,

I am trying to figure out how kqueue/kevent works in a multi-threaded
environment. So I have created a small test program
(http://svn.cmeerw.net/src/nginetd/trunk/test/kqtest.cc) that creates
a socketpair and send 4-byte messages between the 2 sockets. There is
only ever 1 message in transit, but there can be multiple threads
waiting for event notification using kevent.

Note that I have only been able to test this inside a KVM virtual
machine with FreeBSD 9.0 on a dual-core AMD64 laptop.

But what I find extremely interesting is that the time seems to
increase exponentially with the number of worker threads, e.g. I get
(the first argument is the number of worker threads and the second is
the number of iterations):

$ ./kqtest 1 100000
0.508513
$ ./kqtest 2 100000
1.377444
$ ./kqtest 4 100000
2.533485
$ ./kqtest 8 100000
12.305741

BTW, I have also tested in with NetBSD 6.0 BETA on the same machine
(also inside a KVM virtual machine) and there the results look
different:

$ ./kqtest 1 100000
0.480187
$ ./kqtest 2 100000
1.875658
$ ./kqtest 4 100000
1.552574
$ ./kqtest 8 100000
2.024548
$ ./kqtest 16 100000
1.951622

While there is a big difference on NetBSD between 1 and 2 worker
threads, the run time then is roughly stable.

There is also a Linux version (using epoll) of this test program
(http://svn.cmeerw.net/src/nginetd/trunk/test/eptest.cc). There the
timings are much more stable (but this is the actual host operating
system, so no virtual machine) although the single-threaded case is
actually slower than on NetBSD/FreeBSD:

$ ./eptest 1 100000
0.635127
$ ./eptest 2 100000
0.827467
$ ./eptest 4 100000
0.909760
$ ./eptest 8 100000
0.929008
$ ./eptest 16 100000
1.010422


So, I guess my first question is: is this just caused by running it
inside a KVM virtual machine or could someone be so nice and try to
reproduce this behaviour on actual hardware?

Am I doing something completely stupid in the source code? As far as I
can tell, when data is available on one of the sockets, only one
threads gets woken up (so at least from the user space side the
behaviour looks reasonable).


Christof

-- 

http://cmeerw.org                              sip:cmeerw at cmeerw.org
mailto:cmeerw at cmeerw.org                   xmpp:cmeerw at cmeerw.org