Seeing ENOTCAPABLE from write FDs in kqueue?

[Adrian Chadd]

ping?


-a


On 2 June 2014 10:58, Adrian Chadd <adrian at freebsd.org> wrote:
> Hi guys/all,
>
> I've been tinkering with my RSS stuff where I have multiple listen
> sockets, one per worker thread, all terminating high throughput TCP
> transactions. It's all HTTP; I'm using libevent, evhttp and a very
> small amount of glue code to achieve this.
>
> libevent craps out from time to time because occasionally one of the
> events in kqueue returns ENOTCAPABLE.
>
> ENOTCAPABLE: ident=1686, filter=-2, flags=0x00004000, fflags=0x00000000, data=93
>
> ENOTCAPABLE: ident=1324, filter=-2, flags=0x00004000, fflags=0x00000000, data=93
>
> ENOTCAPABLE: ident=1740, filter=-2, flags=0x00004000, fflags=0x00000000, data=93
>
> ENOTCAPABLE: ident=1628, filter=-2, flags=0x00004000, fflags=0x00000000, data=93
>
> ENOTCAPABLE: ident=1199, filter=-2, flags=0x00004000, fflags=0x00000000, data=93
>
> ENOTCAPABLE: ident=818, filter=-2, flags=0x00004000, fflags=0x00000000, data=93
>
> ENOTCAPABLE: ident=389, filter=-2, flags=0x00004000, fflags=0x00000000, data=93
>
> It's happening on the various data FDs; not on the listen socket.
>
> What I'm seeing from ktrace:
>
>  27770 rss-http CAP   operation requires <CAP_EVENT>, descriptor holds
> <CAP_MAC_GET,CAP_MAC_SET,CAP_SEM_GETVALUE,CAP_SEM_POST,CAP_SEM_WAIT,CAP_EVENT,CAP_KQUEUE_EVENT,CAP_IOCTL,CAP_TTYHOOK,CAP_PDGETPID,CAP_PDWAIT,CAP_PDKILL,CAP_EXTATTR_DELETE,CAP_EXTATTR_GET,CAP_EXTATTR_LIST,CAP_EXTATTR_SET,CAP_ACL_CHECK,CAP_ACL_DELETE,CAP_ACL_GET,CAP_ACL_SET,CAP_KQUEUE_CHANGE>
>
>  27770 rss-http CAP   operation requires <CAP_EVENT>, descriptor holds <>
>
>  27770 rss-http CAP   operation requires <CAP_EVENT>, descriptor holds <>
>
>  27770 rss-http CAP   operation requires <CAP_EVENT>, descriptor holds <>
>
>  27770 rss-http CAP   operation requires <CAP_EVENT>, descriptor holds <>
>
>  27770 rss-http CAP   operation requires <CAP_EVENT>, descriptor holds <>
>
> .. so, why exactly would I be seeing this? Is this some capability
> race where we have an FD setup but no capability yet assigned when
> it's added into the kqueue set?
>
> It's happening under high throughput (> 30,000 TCP sessions a second.)
>
> Where would I continue debugging this?
>
> Thanks!
>
>
> -a