Seeing ENOTCAPABLE from write FDs in kqueue?

[Adrian Chadd]

Hi guys/all,

I've been tinkering with my RSS stuff where I have multiple listen
sockets, one per worker thread, all terminating high throughput TCP
transactions. It's all HTTP; I'm using libevent, evhttp and a very
small amount of glue code to achieve this.

libevent craps out from time to time because occasionally one of the
events in kqueue returns ENOTCAPABLE.

ENOTCAPABLE: ident=1686, filter=-2, flags=0x00004000, fflags=0x00000000, data=93

ENOTCAPABLE: ident=1324, filter=-2, flags=0x00004000, fflags=0x00000000, data=93

ENOTCAPABLE: ident=1740, filter=-2, flags=0x00004000, fflags=0x00000000, data=93

ENOTCAPABLE: ident=1628, filter=-2, flags=0x00004000, fflags=0x00000000, data=93

ENOTCAPABLE: ident=1199, filter=-2, flags=0x00004000, fflags=0x00000000, data=93

ENOTCAPABLE: ident=818, filter=-2, flags=0x00004000, fflags=0x00000000, data=93

ENOTCAPABLE: ident=389, filter=-2, flags=0x00004000, fflags=0x00000000, data=93

It's happening on the various data FDs; not on the listen socket.

What I'm seeing from ktrace:

 27770 rss-http CAP   operation requires <CAP_EVENT>, descriptor holds
<CAP_MAC_GET,CAP_MAC_SET,CAP_SEM_GETVALUE,CAP_SEM_POST,CAP_SEM_WAIT,CAP_EVENT,CAP_KQUEUE_EVENT,CAP_IOCTL,CAP_TTYHOOK,CAP_PDGETPID,CAP_PDWAIT,CAP_PDKILL,CAP_EXTATTR_DELETE,CAP_EXTATTR_GET,CAP_EXTATTR_LIST,CAP_EXTATTR_SET,CAP_ACL_CHECK,CAP_ACL_DELETE,CAP_ACL_GET,CAP_ACL_SET,CAP_KQUEUE_CHANGE>

 27770 rss-http CAP   operation requires <CAP_EVENT>, descriptor holds <>

 27770 rss-http CAP   operation requires <CAP_EVENT>, descriptor holds <>

 27770 rss-http CAP   operation requires <CAP_EVENT>, descriptor holds <>

 27770 rss-http CAP   operation requires <CAP_EVENT>, descriptor holds <>

 27770 rss-http CAP   operation requires <CAP_EVENT>, descriptor holds <>

.. so, why exactly would I be seeing this? Is this some capability
race where we have an FD setup but no capability yet assigned when
it's added into the kqueue set?

It's happening under high throughput (> 30,000 TCP sessions a second.)

Where would I continue debugging this?

Thanks!


-a