Kqueue and threading

[Willem Jan Withagen]

On 3-10-2016 19:48, Eric Badger wrote:
> On 10/01/2016 13:02, Willem Jan Withagen wrote:
>> Hi,
>> 
>> Ceph uses a lot of threading, and for any part of it
>> communication it uses epoll(), which FreeBSD does not use. For
>> that there was already a EvenKqueue implementation.
>> 
>> But I think I'm now running into: The kqueue() system call
>> creates a new kernel event queue and returns a descriptor.  The
>> queue is not inherited by a child created with fork(2). However,
>> if rfork(2) is called without the RFFDG flag, then the descriptor
>> table is shared, which will allow sharing of the kqueue between
>> two processes.
>> 
>> Kqueue descriptors are created and events are added, but then
>> the process starts other threads and expects the kqueue-id to be
>> valid there.
>> 
>> However adding more events returns an error, also waiting on the
>> ID for events to happen returns an error (descriptor invalid)
>> 
>> Threading is done with 2 different constructions: std::thread and
>> creating Workers
>> 
>> Would any of these qualify with the quoted warning? and
>> invalidate the descriptor table?
>> 
>> If not, how can I (easily) debug the reason why my descriptors go
>> invalid?
>> 
> 
> Sharing a kqueue between threads of a process works. Are the
> workers created  using rfork without RFFDG as suggested in the
> manpage? I've never had reason to do this, but a quick test seems
> to indicate that it works as advertised. A normal fork closes the
> kqueue file descriptor. If you suspect that's what's happening, you
> might run "procstat -f {worker pid}" to see if file descriptors
> with a "k" (kqueue) in the "T" (type) column appear (if not, they
> were presumably closed on the fork).

Yup, I know what the man page writes. :) But now the trick question
is: how does the C++ worker/thread library function....

Some of the code actually does: std:move on a lambda-function, which
should instantiate a complete fresh environment. And that would
certainly qualify for expected closure of the kq descriptors.
OTOH it keeps regular file descriptors available in the thread.

I guess I'll have to start testing with a few simple C++ programs to
get things clear.

--WjW