How thread-friendly is kevent?
kmacy at freebsd.org
Wed Nov 12 22:14:07 UTC 2014
This may not be related, but it's interesting: a client I worked for
many years ago switched from kqueue to poll for their web server
because kqueue did not distribute the workload evenly. I imagine this
On Wed, Nov 12, 2014 at 12:49 AM, John-Mark Gurney <jmg at funkthat.com> wrote:
> J David wrote this message on Mon, Nov 10, 2014 at 22:00 -0500:
>> On Mon, Nov 10, 2014 at 2:13 AM, John-Mark Gurney <jmg at funkthat.com> wrote:
>> > you
>> > really need to use one of _DISPATCH or _ONESHOT to ensure that the
>> > event only gets delivered to a single thread....
>> That's what one would expect, which is why the observed behavior was
>> so surprising. After increasing the testing load considerably, it did
>> behave as expected (waking more than one thread for one event). But
>> even so, the occurrences were very rare. It would wake up at most one
>> "extra" thread in slightly less than 1 out of 100,000 events.
> This is odd... I would expect that the event w/o _ONESHOT and _DISPATCH
> to be delivered many times... Is it possible you have locks in your
> userland side of things that make this less likely?
>> > Though if you mean how many threads will be woken up in the kernel
>> > and find that there are no events remaining as one of the other kernel
>> > threads has delivered the event, then yes, I have looked at the code,
>> > and there will be a thundering herd problem...
>> Thanks for that, that's exactly the kind of information I was hoping to find.
>> Is that something that can happen without any usermode-visible
>> effects? I.e. all the threads wake up, but they almost all go back to
>> sleep without leaving the kevent() syscall since they can see there's
>> nothing to do anymore. If so, that would match the observed behavior,
>> but could add up to a lot of hidden overhead.
> I have an idea that should only be a few lines of changes that would
> prevent all the threads waking up... As we lock the kq before doing
> the wakeup, we can change KQ_SLEEP from a flag to a count for how many
> threads are sleeping for an event, and if non-zero, do a wakeup_one...
> Then when kqueue_scan is about to exit, check to see if there are
> still events and threads waiting, and then do another wakeup_one...
> Currently, KQ_SLEEP is only a flag, so we have to do wakeup to make
> sure everyone wakes up...
> Well, if you don't have _ONESHOT and _DISPATCH, any changes I make
> should make it more reliable that all threads get the events dispatched
> to them... :)
>> > And if you do, it would make more sense to
>> > use the recent RSS work that Adrian has been working on, and have one
>> > kq per CPU w/ the proper cpu binding for that set of sockets...
>> The most recent information I was able to find:
>> suggests that this work, while admirable and important, is quite some
>> ways away from being production-stable for usermode code:
>> "hopefully I can get my network / rss library up and running enough to
>> prototype an RSS-aware memcached and see if it'll handle this
>> particular workload."
>> It's definitely something to keep an eye on, but probably not a viable
>> approach for us right now.
> True... But some of this is making sure you only run enough threads as
> necessary... As the kq lock is a single lock, having extra threads that
> don't really do much work only increasing contention and other issues...
> John-Mark Gurney Voice: +1 415 225 5579
> "All that I will do, has been done, All that I have, has not."
> freebsd-hackers at freebsd.org mailing list
> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org"
More information about the freebsd-questions