A reliable port cross-build failure (hangup) in my context (amd64->armv7 cross build, with native-tool speedup involved) [details of a specific qemu-arm-static source code problem]

Mark Millard marklmi at yahoo.com
Mon Dec 31 00:38:57 UTC 2018

On 2018-Dec-28, at 12:12, Mark Millard <marklmi at yahoo.com> wrote:

> On 2018-Dec-28, at 05:13, Michal Meloun <melounmichal at gmail.com> wrote:
>> Mark,
>> this is known problem with qemu-user-static.
>> Emulation of every single interruptible syscall is broken by design (it
>> have signal related races). Theses races cannot be solved without major
>> rewrite of syscall emulation code.
>> Unfortunately, nobody actively works on this, I think.
> Thanks for the note setting some expectations.
> . . .

It turns out that I've been through (part of?) this before and
mikael.urankar at gmail.com had back then provided a qemu-user-static
patch (that might have been arm specific or 32-bit target specific
when running on a 64-bit host). (The qemu-user-static code structure
seems to have changed some afterwards and the patch is no longer
where he had pointed me to back then.)

To show size and offsets on armv7 vs. armd64 for struct kevent
I use:

# more kevent_size_offsets.c 
#include "/usr/include/sys/event.h" // kevent
#include <stddef.h> // offsetof
#include <stdio.h>  // printf

        printf("%lu\n", (unsigned long) sizeof(struct kevent));
        printf("ident %lu\n", (unsigned long) offsetof(struct kevent, ident));
        printf("filter %lu\n", (unsigned long) offsetof(struct kevent, filter));
        printf("flags %lu\n", (unsigned long) offsetof(struct kevent, flags));
        printf("fflags %lu\n", (unsigned long) offsetof(struct kevent, fflags));
        printf("data %lu\n", (unsigned long) offsetof(struct kevent, data));
        printf("udata %lu\n", (unsigned long) offsetof(struct kevent, udata));
        printf("ext %lu\n", (unsigned long) offsetof(struct kevent, ext));
        return 0;

It ends up showing on armv7 (under qemu-arm-static insteead of native, not
that it matters here):

# ./a.out
ident 0
filter 4
flags 6
fflags 8
data 16
udata 24
ext 32

On amd64 (native) it ends up as:

# ./a.out
ident 0
filter 8
flags 10
fflags 12
data 16
udata 24
ext 32

Thus a translation of layout is required when hosted. This is for:

struct kevent {
        __uintptr_t     ident;          /* identifier for this event */
        short           filter;         /* filter for event */
        unsigned short  flags;          /* action flags for kqueue */
        unsigned int    fflags;         /* filter flag value */
        __int64_t       data;           /* filter data value */
        void            *udata;         /* opaque user data identifier */
        __uint64_t      ext[4];         /* extensions */

But qemu-user-static has for translation purposes:

struct target_freebsd_kevent {
    abi_ulong  ident;
    int16_t    filter;
    uint16_t   flags;
    uint32_t   fflags;
    int64_t data;
    abi_ulong  udata;
    uint64_t  ext[4];
} __packed;

(note the __packed) for which in amd64's qemu_arm_static has
the size and offsets:

# gdb qemu-arm-static
. . .
(gdb) p/d sizeof(struct target_freebsd_kevent)
$1 = 56
(gdb) p/d &((struct target_freebsd_kevent *)0)->ident
$2 = 0
(gdb) p/d &((struct target_freebsd_kevent *)0)->filter
$3 = 4
(gdb) p/d &((struct target_freebsd_kevent *)0)->flags
$4 = 6
(gdb) p/d &((struct target_freebsd_kevent *)0)->fflags
$5 = 8
(gdb) p/d &((struct target_freebsd_kevent *)0)->data
$6 = 12
(gdb) p/d &((struct target_freebsd_kevent *)0)->udata
$7 = 20
(gdb) p/d &((struct target_freebsd_kevent *)0)->ext
$8 = 24

which which does not match the armv7 offsets for
data, udata, or ext and does not have the right size
for struct target_freebsd_kevent[] indexing to
match armv7's struct target_freebsd_kevent[] indexing.

This in turn makes the do_freebsd_kevent code do the wrong
thing in its:

    struct target_freebsd_kevent *target_changelist, *target_eventlist;
. . .
        for (i = 0; i < arg3; i++) {
            __get_user(changelist[i].ident, &target_changelist[i].ident);
            __get_user(changelist[i].filter, &target_changelist[i].filter);
            __get_user(changelist[i].flags, &target_changelist[i].flags);
            __get_user(changelist[i].fflags, &target_changelist[i].fflags);
            __get_user(changelist[i].data, &target_changelist[i].data);
            /* __get_user(changelist[i].udata, &target_changelist[i].udata); */
            changelist[i].udata = (void *)(uintptr_t)target_changelist[i].udata;
            tswap32s((uint32_t *)&changelist[i].udata);
            changelist[i].udata = (void *)(uintptr_t)target_changelist[i].udata;
            tswap64s((uint64_t *)&changelist[i].udata);
            __get_user(changelist[i].ext[0], &target_changelist[i].ext[0]);
            __get_user(changelist[i].ext[1], &target_changelist[i].ext[1]);
            __get_user(changelist[i].ext[2], &target_changelist[i].ext[2]);
            __get_user(changelist[i].ext[3], &target_changelist[i].ext[3]);
. . .
        for (i = 0; i < arg5; i++) {
            __put_user(eventlist[i].ident, &target_eventlist[i].ident);
            __put_user(eventlist[i].filter, &target_eventlist[i].filter);
            __put_user(eventlist[i].flags, &target_eventlist[i].flags);
            __put_user(eventlist[i].fflags, &target_eventlist[i].fflags);
            __put_user(eventlist[i].data, &target_eventlist[i].data);
            /* __put_user(eventlist[i].udata, &target_eventlist[i].udata);*/
            tswap32s((uint32_t *)&eventlist[i].data);
            target_eventlist[i].data = (uintptr_t)eventlist[i].data;
            tswap64s((uint64_t *)&eventlist[i].data);
            target_eventlist[i].data = (uintptr_t)eventlist[i].data;
            __put_user(eventlist[i].ext[0], &target_eventlist[i].ext[0]);
            __put_user(eventlist[i].ext[1], &target_eventlist[i].ext[1]);
            __put_user(eventlist[i].ext[2], &target_eventlist[i].ext[2]);
            __put_user(eventlist[i].ext[3], &target_eventlist[i].ext[3]);

I'll eventually do something to have struct target_freebsd_kevent for
amd64-native targeting armv7 and see if that is sufficient to avoid the
problem in my context. Previously removing the __packed was enough to
make the structure the same size with the same offsets as for armv7.
(Such might not have been appropriate to all targets.)

armv6 would have the same problem as I understand things.

Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

More information about the freebsd-current mailing list