Kqueue and threading

[Willem Jan Withagen]

On 3-10-2016 19:48, Eric Badger wrote:
> On 10/01/2016 13:02, Willem Jan Withagen wrote:
>> Hi,
>>
>> Ceph uses a lot of threading, and for any part of it communication it
>> uses epoll(), which FreeBSD does not use. For that there was already a
>> EvenKqueue implementation.
>>
>> But I think I'm now running into:
>>      The kqueue() system call creates a new kernel event queue and
>>      returns a descriptor.  The queue is not inherited by a child
>>      created with fork(2). However, if rfork(2) is called without the
>>      RFFDG flag, then the descriptor table is shared, which will allow
>>      sharing of the kqueue between two processes.
>>
>> Kqueue descriptors are created and events are added, but then the
>> process starts other threads and expects the kqueue-id to be valid there.
>>
>> However adding more events returns an error, also waiting on the ID for
>> events to happen returns an error (descriptor invalid)
>>
>> Threading is done with 2 different constructions:
>> 	std::thread
>> and 	creating Workers
>>
>> Would any of these qualify with the quoted warning? and invalidate the
>> descriptor table?
>>
>> If not, how can I (easily) debug the reason why my descriptors go invalid?
>>
> 
> Sharing a kqueue between threads of a process works. Are the workers
> created  using rfork without RFFDG as suggested in the manpage? I've
> never had reason to do this, but a quick test seems to indicate that it
> works as advertised. A normal fork closes the kqueue file descriptor. If
> you suspect that's what's happening, you might run "procstat -f {worker
> pid}" to see if file descriptors with a "k" (kqueue) in the "T" (type)
> column appear (if not, they were presumably closed on the fork).

Right,

This has become a rather serious and painful undertaking, but it seems
to work now, and all tests seem to be working oke ....

But getting at the workers creation is rather impossible since that is
mostly hidden in some Boost libs...

Fortunate the systemcode for events is rather fleshed out into modules
and there is a EventKqueue class, that hides all gory details.
So there I main lists of what events are being set, and once I detect a
change of thread, I reinstall the list with events. But that is
obviously rather cumbersome.

--WjW



So what I do know