Using kqueue with aio_read/write

[Willem Jan Withagen]

On 28-12-2018 02:47, Alan Somers wrote:
> On Thu, Dec 27, 2018 at 6:15 PM Willem Jan Withagen <wjw at digiware.nl> wrote:
>>
>> Hi,
>>
>> Im trying to understand why I cannot get so code to work.
>> This is the smallest extract I can make to show my problem.
>>
>> I would expect the kevent() call to return every timeo tick.
>> Even if I tell it NOT to time-out I get these spurts of errors
>>
>> Since there is nothing to trigger the AIO-event, I would expect kqueue
>> to hold indefinitly.
>>
>> But it does not generate anything other than errors
>> And instead it repeatedly complains that there is a permission error:
>>     get_events_kevent: EV_Error(1) kevent(): Operation not permitted
>>
>> But I'm not getting where that would the case...
>>
>> Surely a pilot error, but I do overlook it al the time.
>> So suggestions are welcome.
>>
>> Thanx,
>> --WjW
>>
>> #include <aio.h>
>> #include <errno.h>
>> #include <fcntl.h>
>> #include <stdio.h>
>> #include <stdlib.h>
>> #include <string.h>
>> #include <sys/stat.h>
>> #include <sys/event.h>
>> #include <unistd.h>
>>
>> #define BUFFER_SIZE     512
>> #define MAX_EVENTS 32
>>
>> #define FILENAME "/tmp/aio_test"
>> char filename[256];
>> int fd;
>> int done = 0;
>>
>> void get_events_kevent(int fd, int kq)
>> {
>>       printf("get_events function fd = %d, kq = %d\n", fd, kq);
>>       int i = 0, errcnt = 0, err, ret, reterr, rev;
>>       int search = 1;
>>
>>       int timeout_ms = 10;
>>       struct timespec timeo = {
>>           timeout_ms / 1000,
>>           (timeout_ms % 1000) * 1000 * 1000
>>       };
>>       struct kevent filter[16];
>>       struct kevent changed[16];
>>
>>       EV_SET(&filter[0], fd, EVFILT_AIO,
>>               EV_ADD,
>>               0, 0, 0 );
> 
> 
> This is the first problem.  There's no need to explicitly set
> EVFILT_AIO on the kqueue.  It gets set by the aio_read(2) or similar
> syscall.  And this invocation wouldn't be correct anyway, because for
> AIO the ident field refers to the address of the struct aiocb, not the
> file descriptor.  If the only events you care about are AIO, then you
> can pass NULL as the filter argument to kevent.  I suspect this is the
> cause of your problem.  The kernel probably thinks you're trying to
> register for an aiocb that's outside of your address space or
> something like that.

Hi Alan,

That at least helps against getting EPERM.
And I get a nice stream of timeouts hitting kevent.

It sort of makes sense when you write it like this, but then this is not 
clear at all from the man pages (for me that is)
But it seems sort of weird to "not filter" to get AIO events.

>>       while (!done) {
>>           printf("+"); 
>>           rev = kevent(kq, filter, 1, changed, 16, 0); //&timeo);
>>           if (rev < 0) {
>>               perror("kevent error");
>>           } else if (rev == 0) {
>>               printf("T");
>>           } else {
>>               printf("rev(%d)\n", rev);
>>               if (changed[0].flags == EV_ERROR) {
>>                   errno = changed[0].data;
>>                   printf( "%s: EV_Error(%d) kevent(): %s\n", __func__, errno,
>>                       strerror(errno));
>>                   memset(&changed[0], 0, sizeof(struct kevent));
>>               } else {
>>                   err = aio_error((struct aiocb*)changed[0].udata);
> 
> 
> No need to call aio_error(2) after kevent(2) returns.  You can go
> straight to aio_return.  aio_error shouldn't hurt, but it isn't
> necessary.

This is a sort of leftover from earlier trials where I was using a "home 
made" Poll iterating over the aiocb blocks.

Thanx for the help, now I can go on trying getting this into Ceph.
Perhaps I'll try an clean this up, and post a working example somewhere 
online.

--WjW

> 
> 
>>                   ret = aio_return((struct aiocb*)changed[0].udata);
>>                   if (ret < 0 )
>>                       reterr = errno;
>>                   if (err != 0) {
>>                       printf( "%s: slot: %d, Error(%d) at aio_error():
>> %s\n", __func__, i, err, strerror (err));
>>                       errcnt++;
>>                       if (errcnt > 50) {
>>                           exit(3);
>>                       }
>>                   }
>>               }
>>           }
>>       }
>> }
>>
>> int main()
>> {
>>       (void) strcpy(filename, FILENAME);
>>       unlink(filename);
>>       fd = open(filename, O_CREAT | O_RDWR, S_IRUSR | S_IWUSR);
>>       if (fd == -1) {
>>           printf( "%s: Error at open(): %s\n", __func__, strerror(errno));
>>           exit(1);
>>       }
>>
>>       int kq = kqueue();
>>       if (fd == -1) {
>>           printf( "%s: Error at kqueue(): %s\n", __func__, strerror(errno));
>>           exit(1);
>>       }
>>
>>       get_events_kevent( fd, kq);
>>
>>       close(kq);
>>       close(fd);
>>       return 0;
>> }
>>
>>
>> _______________________________________________
>> freebsd-hackers at freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
>> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org"