How thread-friendly is kevent?

[K. Macy]

This may not be related, but it's interesting: a client I worked for
many years ago switched from kqueue to poll for their web server
because kqueue did not distribute the workload evenly. I imagine this
is unchanged.

-K

On Wed, Nov 12, 2014 at 12:49 AM, John-Mark Gurney <jmg at funkthat.com> wrote:
> J David wrote this message on Mon, Nov 10, 2014 at 22:00 -0500:
>> On Mon, Nov 10, 2014 at 2:13 AM, John-Mark Gurney <jmg at funkthat.com> wrote:
>> > you
>> > really need to use one of _DISPATCH or _ONESHOT to ensure that the
>> > event only gets delivered to a single thread....
>>
>> That's what one would expect, which is why the observed behavior was
>> so surprising.  After increasing the testing load considerably, it did
>> behave as expected (waking more than one thread for one event).  But
>> even so, the occurrences were very rare.  It would wake up at most one
>> "extra" thread in slightly less than 1 out of 100,000 events.
>
> This is odd...  I would expect that the event w/o _ONESHOT and _DISPATCH
> to be delivered many times...  Is it possible you have locks in your
> userland side of things that make this less likely?
>
>> > Though if you mean how many threads will be woken up in the kernel
>> > and find that there are no events remaining as one of the other kernel
>> > threads has delivered the event, then yes, I have looked at the code,
>> > and there will be a thundering herd problem...
>>
>> Thanks for that, that's exactly the kind of information I was hoping to find.
>>
>> Is that something that can happen without any usermode-visible
>> effects?  I.e. all the threads wake up, but they almost all go back to
>> sleep without leaving the kevent() syscall since they can see there's
>> nothing to do anymore.  If so, that would match the observed behavior,
>> but could add up to a lot of hidden overhead.
>
> I have an idea that should only be a few lines of changes that would
> prevent all the threads waking up...  As we lock the kq before doing
> the wakeup, we can change KQ_SLEEP from a flag to a count for how many
> threads are sleeping for an event, and if non-zero, do a wakeup_one...
> Then when kqueue_scan is about to exit, check to see if there are
> still events and threads waiting, and then do another wakeup_one...
>
> Currently, KQ_SLEEP is only a flag, so we have to do wakeup to make
> sure everyone wakes up...
>
> Well, if you don't have _ONESHOT and _DISPATCH, any changes I make
> should make it more reliable that all threads get the events dispatched
> to them... :)
>
>> > And if you do, it would make more sense to
>> > use the recent RSS work that Adrian has been working on, and have one
>> > kq per CPU w/ the proper cpu binding for that set of sockets...
>>
>> The most recent information I was able to find:
>>
>> http://adrianchadd.blogspot.com/2014/10/more-rss-udp-tests-this-time-on-dell.html
>>
>> suggests that this work, while admirable and important, is quite some
>> ways away from being production-stable for usermode code:
>>
>> "hopefully I can get my network / rss library up and running enough to
>> prototype an RSS-aware memcached and see if it'll handle this
>> particular workload."
>>
>> It's definitely something to keep an eye on, but probably not a viable
>> approach for us right now.
>
> True... But some of this is making sure you only run enough threads as
> necessary...  As the kq lock is a single lock, having extra threads that
> don't really do much work only increasing contention and other issues...
>
> --
>   John-Mark Gurney                              Voice: +1 415 225 5579
>
>      "All that I will do, has been done, All that I have, has not."
> _______________________________________________
> freebsd-hackers at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org"