Kqueue and threading

[Eric Badger]

On 10/01/2016 13:02, Willem Jan Withagen wrote:
> Hi,
>
> Ceph uses a lot of threading, and for any part of it communication it
> uses epoll(), which FreeBSD does not use. For that there was already a
> EvenKqueue implementation.
>
> But I think I'm now running into:
>      The kqueue() system call creates a new kernel event queue and
>      returns a descriptor.  The queue is not inherited by a child
>      created with fork(2). However, if rfork(2) is called without the
>      RFFDG flag, then the descriptor table is shared, which will allow
>      sharing of the kqueue between two processes.
>
> Kqueue descriptors are created and events are added, but then the
> process starts other threads and expects the kqueue-id to be valid there.
>
> However adding more events returns an error, also waiting on the ID for
> events to happen returns an error (descriptor invalid)
>
> Threading is done with 2 different constructions:
> 	std::thread
> and 	creating Workers
>
> Would any of these qualify with the quoted warning? and invalidate the
> descriptor table?
>
> If not, how can I (easily) debug the reason why my descriptors go invalid?
>

Sharing a kqueue between threads of a process works. Are the workers
created  using rfork without RFFDG as suggested in the manpage? I've
never had reason to do this, but a quick test seems to indicate that it
works as advertised. A normal fork closes the kqueue file descriptor. If
you suspect that's what's happening, you might run "procstat -f {worker
pid}" to see if file descriptors with a "k" (kqueue) in the "T" (type)
column appear (if not, they were presumably closed on the fork).

Eric

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 632 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20161003/528e7d50/attachment.sig>