head -r339076 amd64 -> armv7 port cross build attempt with native tools involved: hangs between a cc (wait) and its child ld (uwait)

Sat Oct 27 05:45:52 UTC 2018

[Attaching to the ld process with gdb and detaching let things
continue.]

On 2018-Oct-26, at 8:42 PM, Mark Millard <marklmi at yahoo.com> wrote:

> In trying to amd64 -> armv7 cross build ports via poudriere-devel
> use with native cross tools involved (and UFS, not ZFS), I'm
> getting about 117 ports that built and then one that ends up stuck
> in wait/uwait . ^C to poudriere and restarting it repeats the
> stuck behavior at the same point (a cc and its ld), for example:
> 
> 
> [00:02:51] [01] [00:00:00] Building print/texinfo | texinfo-6.5,1
> 
> ps output extraction (blank lines added for each of
> scanning):
> 
> UID   PID  PPID CPU PRI NI    VSZ   RSS MWCHAN STAT TT     TIME COMMAND
> . . .
>  0 42312 32181   0  52  0  12904  3904 select I     1  0:00.02 sh: poudriere[FBSDFSSDjailArmV7-default][01]: build_pkg (texinfo-6.5,1) (sh)
>  0 42974 42312   0  52  0  12904  3900 wait   I     1  0:00.00 sh: poudriere[FBSDFSSDjailArmV7-default][01]: build_pkg (texinfo-6.5,1) (sh)
>  0 42975 42974   0  52  0  10408  1840 wait   IJ    1  0:00.01 /usr/bin/make -C /usr/ports/print/texinfo configure
>  0 43077 42975   0  52  0  10252  1792 wait   IJ    1  0:00.00 /bin/sh -e -c (cd /wrkdirs/usr/ports/print/texinfo/work/texinfo-6.5 &&  _LATE_CONFIGURE_ARGS="" ;  if [ -z "" ] &&  ./configure --help 
>  0 43375 43077   0  52  0  11164  2392 wait   IJ    1  0:00.19 /bin/sh ./configure --enable-nls --prefix=/usr/local --localstatedir=/var --mandir=/usr/local/man --disable-silent-rules --infodir=/usr
>  0 46850 43375   0  52  0  11164  2388 wait   IJ    1  0:00.00 /bin/sh ./configure --enable-nls --prefix=/usr/local --localstatedir=/var --mandir=/usr/local/man --disable-silent-rules --infodir=/usr
>  0 46857 46850   0  52  0  11080  2060 wait   IJ    1  0:00.04 /bin/sh ./configure --disable-option-checking --prefix=/usr/local --enable-nls --localstatedir=/var --mandir=/usr/local/man --disable-s
> 
>  0 47796 46857   0  52  0 113840 26184 wait   IJ    1  0:00.15 /usr/local/bin/qemu-arm-static /usr/bin/cc -o conftest -O2 -pipe -mcpu=cortex-a7 -DLIBICONV_PLUG -g -fno-strict-aliasing -mcpu=cortex-a
> 
>  0 47801 47796   0  52  0 285300 39672 uwait  IJ    1  0:00.22 qemu-arm-static -L /usr/gnemul/qemu-arm /usr/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-
> 
> So the "/usr/local/bin/qemu-arm-static /usr/bin/cc . . ."
> creates the child "qemu-arm-static -L /usr/gnemul/qemu-arm /usr/bin/ld . . ."
> process and the two get hung up. Letting it sit for long periods does
> not let it progress.
> 
> The full commands are (note the "-pipe" vs. the "/tmp/conftest-6c0832.o"):
> 
> /usr/local/bin/qemu-arm-static /usr/bin/cc -o conftest -O2 -pipe -mcpu=cortex-a7 -DLIBICONV_PLUG -g -fno-strict-aliasing -mcpu=cortex-a7 -DLIBICONV_PLUG conftest.c
> 
> and:
> 
> qemu-arm-static -L /usr/gnemul/qemu-arm /usr/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o conftest /usr/lib/crt1.o /usr/lib/crti.o /usr/lib/crtbegin.o -L/usr/lib /tmp/conftest-6c0832.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/crtend.o /usr/lib/crtn.o
> 
> For reference for /tmp/conftest-6c0832.o :
> 
> # ls -lTd /usr/local/poudriere/data/.m/FBSDFSSDjailArmV7-default/01/tmp/conftest-6c0832.o
> -rw-r--r--  1 root  wheel  4204 Oct 26 17:33:13 2018 /usr/local/poudriere/data/.m/FBSDFSSDjailArmV7-default/01/tmp/conftest-6c0832.o
> 
> (I'm not using tmpfs or the like at all.)
> 
> 
> The context is based on head -r339076 an is on a
> Ryzen Threadripper 1950X system, natively booted
> (not Hyper-V). (I've not tried under Hyper-V yet.)
> 
> Note: I have built ports similarly before --but the
> last time was back in March-May sometime.
> 
> # poudriere jail -jFBSDFSSDjailArmV7 -i
> Jail name:         FBSDFSSDjailArmV7
> Jail version:      12.0-ALPHA8
> Jail arch:         arm.armv7
> Jail method:       null
> Jail mount:        /usr/obj/DESTDIRs/clang-armv7-installworld-poud
> Jail fs:           
> Jail updated:      2018-10-26 16:42:55
> Tree name:         default
> Tree method:       null
> Status:            parallel_build:
> Building started:  2018-10-26 17:29:36
> Elapsed time:      02:47:50
> Packages built:    0
> Packages failed:   0
> Packages ignored:  0
> Packages skipped:  0
> Packages total:    84
> Packages left:     84
> 
> # poudriere ports -l
> PORTSTREE METHOD TIMESTAMP           PATH
> default   null   2017-08-14 21:07:05 /usr/ports
> 
> 
> I have yet to think of a way to look into this or to work around
> it. But my long running build on an Orange Pi Plus 2nd Edition
> has finished so I'll update from that for now.

I tried again and when it hung up I used gdb to
attach to the ld process and later to detach:

# gdb `which qemu-arm-static`
. . .
(gdb) attach 18703
Attaching to program: /usr/local/bin/qemu-arm-static, process 18703
Couldn't get registers: Device busy.
. . .
(gdb) bt
#0  _umtx_op () at _umtx_op.S:3
#1  0x0000000060050cd4 in _umtx_wait_uint_private (where=0x0, addr=<optimized out>, target_val=<optimized out>, tsz=<optimized out>, t=<optimized out>)
    at /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-495fb3a/bsd-user/freebsd/os-thread.c:258
#2  freebsd_lock_umutex (target_addr=4102556064, id=100867, ts=0x0, mode=<optimized out>) at /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-495fb3a/bsd-user/freebsd/os-thread.c:890
#3  0x000000006004a808 in do_freebsd__umtx_op (obj=4102556064, op=<optimized out>, val=0, uaddr=0, target_time=0)
    at /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-495fb3a/bsd-user/freebsd/os-thread.h:359
#4  0x00000000600414d5 in do_freebsd_syscall (cpu_env=0x8607a4c58, num=454, arg1=<optimized out>, arg2=<optimized out>, arg3=<optimized out>, arg4=0, arg5=0, arg6=-185272152, arg7=0, arg8=0)
    at /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-495fb3a/bsd-user/syscall.c:1364
#5  0x0000000060038d03 in target_cpu_loop (env=0x8607a4c58) at /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-495fb3a/bsd-user/arm/target_arch_cpu.h:207
#6  0x00000000600386a9 in cpu_loop (env=0xf48809bc) at /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-495fb3a/bsd-user/main.c:121
#7  0x0000000060039922 in main (argc=-10608, argv=0x7fffffffd1d8) at /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-495fb3a/bsd-user/main.c:513
(gdb) detach
Detaching from program: /usr/local/bin/qemu-arm-static, process 18703

Things started back up from there.

We will see if it hangs up again.

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)