How thread-friendly is kevent?

[John-Mark Gurney]

J David wrote this message on Sun, Nov 09, 2014 at 14:57 -0500:
> How thread-friendly is kevent()?
> 
> In most cases, a dedicated thread does the kevent()-ing and
> dispatching work to the thread pool, but at extremely high rates that
> thread becomes a significant bottleneck.
> 
> As an example, consider a pool of, say, 17 threads on a 16 core
> machine all in kevent() waiting for one of 32000 open TCP connections
> to be read-ready.  One connection becomes read-ready.  How many
> threads will have kevent() return with that event in eventlist?  Is
> there potential for a thundering herd problem?

It depends... kevent is a level triggered system...  By default, if
you don't use _ONESHOT or _DISPATCH, all threads will be woken up..
So, yes, there is potential for a thundering herd problem...

> Limited small-scale experimentation suggests that only one thread
> returns per event, but it's not documented that way, so it's not clear
> if that behavior is intended, an implementation detail, or a
> coincidence that won't hold up at scale.
> 
> Is this behavior at all guaranteed / by design / intentional?
> 
> Although it would be ideal if so, it would also make sense to have to
> rely on EV_DISPATCH in multithreaded applications to prevent events
> from being delivered more than once, or to use EV_ONESHOT and re-add
> the event entirely, depending on which approach better suits the
> internal data structure the kernel uses for kqueue.

If you do have a single kq w/ multiple threads (you cannot have a
shared kq between processes, as the fork clears out all events), you
really need to use one of _DISPATCH or _ONESHOT to ensure that the
event only gets delivered to a single thread....

Though if you mean how many threads will be woken up in the kernel
and find that there are no events remaining as one of the other kernel
threads has delivered the event, then yes, I have looked at the code,
and there will be a thundering herd problem...

This could be mitigated with out too much work, though the question I
have is why do you have so many threads listening on the same kqueue?

The most common use of this is for socket IO (there isn't much else
except maybe vnodes) that you can wait on that you'd have such a highly
multithreaded program...  And if you do, it would make more sense to
use the recent RSS work that Adrian has been working on, and have one
kq per CPU w/ the proper cpu binding for that set of sockets...

-- 
  John-Mark Gurney				Voice: +1 415 225 5579

     "All that I will do, has been done, All that I have, has not."