FYI: ports head -r484783 poudriere-devel with qemu-arm-static: sometimes hangs between a cc (wait) and its child ld (uwait)
Mark Millard
marklmi at yahoo.com
Tue Nov 13 08:42:44 UTC 2018
On 2018-Nov-12, at 10:18, Mark Millard <marklmi at yahoo.com> wrote:
> On 2018-Nov-12, at 05:54, Kyle Evans <kevans at freebsd.org> wrote:
>
>> On Sun, Nov 11, 2018 at 9:11 PM Mark Millard <marklmi at yahoo.com> wrote:
>>>
>>> [I still can not produce the problem below on demand.
>>> It seems racy with no fixed context producing the
>>> problem as far as which port is building. But the
>>> general structure of what hangs is the same each
>>> time so far.]
>>>
>>> The following is just an FYI for the other
>>> qemu-arm-static tied problem that I regularly run into.
>>> I do not have much useful information so far. It is
>>> not clear how I'd get such information.
>>>
>>
>> Hi,
>>
>> Just so we're clear- in what kind of time frame did you start
>> observing this hang?
>
> Unfortunately, I did no qemu-user-static use after
> 2018-Feb-6 until 2018-10-26. My list activity reported
> the problem for the first time on Oct. 26 and I had
> updated before using qemu-arm-static on the 26th.
>
> Looks like back on Feb. 6 I was using: qemu-user-static-2.11.50.g20171215_3
>
> Looks like back on Oct. 26 I was using: qemu-user-static-2.11.50.g20180622_1
>
> I'm now using qemu-user-static-2.11.50.g20181011 .
>
>
> For reference:
>
> The Feb cross build logs for Feb 6 show things like:
>
> =>> Building ports-mgmt/poudriere-devel
> build started at Tue Feb 6 17:39:36 PST 2018
> port directory: /usr/ports/ports-mgmt/poudriere-devel
> package name: poudriere-devel-3.2.99.20180202_2
> building for: FreeBSD FBSDFSSDjailVariant 12.0-CURRENT FreeBSD 12.0-CURRENT r327485M arm
> maintained by: bdrewery at FreeBSD.org
> Makefile ident: $FreeBSD: head/ports-mgmt/poudriere-devel/Makefile 461075 2018-02-06 16:33:15Z brd $
> Poudriere version: 3.2.99.20180202_2
> Host OSVERSION: 1200054
> Jail OSVERSION: 1200054
>
> The amd64 (host) logs before that show for qemu-user-static:
>
> =>> Building emulators/qemu-user-static
> build started at Sun Feb 4 11:22:59 PST 2018
> port directory: /usr/ports/emulators/qemu-user-static
> package name: qemu-user-static-2.11.50.g20171215_3
> building for: FreeBSD FBSDFSSDjailVariant 12.0-CURRENT FreeBSD 12.0-CURRENT r327485M amd64
> maintained by: sbruno at FreeBSD.org
> Makefile ident: $FreeBSD: head/emulators/qemu-user-static/Makefile 441455 2017-05-22 13:17:38Z linimon $
> Poudriere version: 3.2.99.20180202_1
> Host OSVERSION: 1200054
> Jail OSVERSION: 1200054
>
> (I normally keep the system source code the same across TARGET_ARCH's,
> with some exceptions for powerpc families.)
>
> Oct. 26 shows for qemu-user-static:
>
> =>> Building emulators/qemu-user-static
> build started at Fri Oct 26 13:55:50 PDT 2018
> port directory: /usr/ports/emulators/qemu-user-static
> package name: qemu-user-static-2.11.50.g20180622_1
> building for: FreeBSD FBSDFSSDjailVariant 12.0-ALPHA8 FreeBSD 12.0-ALPHA8 #1 r339076:339432M: Mon Oct 22 17:48:28 PDT 2018 markmi at FBSDFSSD:/usr/obj/amd64_clang_alt/amd64.amd64/usr/src/amd64.amd64/sys/GENERIC-NODBG amd64
> maintained by: sbruno at FreeBSD.org
> Makefile ident: $FreeBSD: head/emulators/qemu-user-static/Makefile 441455 2017-05-22 13:17:38Z linimon $
> Poudriere version: 3.2.99.20180511
> Host OSVERSION: 1200084
> Jail OSVERSION: 1200063
>
> The armv7 jail context would also be based on the same system source,
> mostly -r339076 source.
>
>
> Currently for qemu-user-static I'm at:
>
> =>> Building emulators/qemu-user-static
> build started at Sun Nov 11 14:52:52 PST 2018
> port directory: /usr/ports/emulators/qemu-user-static
> package name: qemu-user-static-2.11.50.g20181011
> building for: FreeBSD FBSDFSSDjailVariant 13.0-CURRENT FreeBSD 13.0-CURRENT amd64
> maintained by: sbruno at FreeBSD.org
> Makefile ident: $FreeBSD: head/emulators/qemu-user-static/Makefile 441455 2017-05-22 13:17:38Z linimon $
> Poudriere version: 3.2.99.20181024
> Host OSVERSION: 1300003
> Jail OSVERSION: 1300003
I did some buildworld's inside a bulk -i session.
I got a hangup that was not lld: an emulated
ctfmerge hangup. It too had a thread fanout, but
only 7 threads. The attach/detach sequence did
not start things going. The thread backtraces
look like for the lld example.
(gdb) info threads
Id Target Id Frame
* 1 LWP 100502 of process 64885 _umtx_op () at _umtx_op.S:3
2 LWP 101208 of process 64885 _umtx_op_err () at /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:37
3 LWP 102353 of process 64885 _umtx_op () at _umtx_op.S:3
4 LWP 101899 of process 64885 _umtx_op () at _umtx_op.S:3
5 LWP 101783 of process 64885 _umtx_op () at _umtx_op.S:3
6 LWP 101902 of process 64885 _umtx_op () at _umtx_op.S:3
7 LWP 101907 of process 64885 _umtx_op () at _umtx_op.S:3
(gdb) bt
#0 _umtx_op () at _umtx_op.S:3
#1 0x0000000060050c34 in _umtx_wait_uint_private (addr=<optimized out>, target_val=<optimized out>, tsz=<optimized out>, t=<optimized out>, where=<optimized out>)
at /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/bsd-user/freebsd/os-thread.c:258
#2 freebsd_lock_umutex (target_addr=4139717232, id=100502, ts=0x0, mode=<optimized out>) at /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/bsd-user/freebsd/os-thread.c:890
#3 0x000000006004a768 in do_freebsd__umtx_op (obj=4139717232, op=<optimized out>, val=0, uaddr=0, target_time=0)
at /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/bsd-user/freebsd/os-thread.h:359
#4 0x0000000060041405 in do_freebsd_syscall (cpu_env=0x860596318, num=454, arg1=<optimized out>, arg2=<optimized out>, arg3=<optimized out>, arg4=0, arg5=0, arg6=165556, arg7=100502, arg8=0)
at /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/bsd-user/syscall.c:1364
#5 0x0000000060038bb3 in target_cpu_loop (env=0x860596318) at /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/bsd-user/arm/target_arch_cpu.h:207
#6 0x0000000060038559 in cpu_loop (env=0xf6bf128c) at /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/bsd-user/main.c:121
#7 0x00000000600397d2 in main (argc=-18969, argv=0x7fffffffaea0) at /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/bsd-user/main.c:513
(gdb) thread 2
[Switching to thread 2 (LWP 101208 of process 64885)]
#0 _umtx_op_err () at /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:37
37 RSYSCALL_ERR(_umtx_op)
(gdb) bt
#0 _umtx_op_err () at /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:37
#1 0x00000000601a3cc1 in _thr_umtx_timedwait_uint (mtx=0x8604de008, id=<optimized out>, clockid=<optimized out>, abstime=<optimized out>, shared=<optimized out>)
at /usr/src/lib/libthr/thread/thr_umtx.c:236
#2 0x00000000601ad548 in cond_wait_user (abstime=0x0, cancel=1, cvp=<optimized out>, mp=<optimized out>) at /usr/src/lib/libthr/thread/thr_cond.c:307
#3 cond_wait_common (cond=<optimized out>, mutex=<optimized out>, abstime=0x0, cancel=1) at /usr/src/lib/libthr/thread/thr_cond.c:367
#4 0x00000000601143cc in qemu_futex_wait (val=4294967295, ev=<optimized out>) at util/qemu-thread-posix.c:350
#5 qemu_event_wait (ev=0x626eeec0 <rcu_call_ready_event>) at util/qemu-thread-posix.c:445
#6 0x000000006011b3ba in call_rcu_thread (opaque=<optimized out>) at util/rcu.c:255
#7 0x00000000601ad1c6 in thread_start (curthread=0x8604d2500) at /usr/src/lib/libthr/thread/thr_create.c:292
#8 0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x7fffdfffd000
(gdb) thread 3
[Switching to thread 3 (LWP 102353 of process 64885)]
#0 _umtx_op () at _umtx_op.S:3
3 RSYSCALL(_umtx_op)
(gdb) bt
#0 _umtx_op () at _umtx_op.S:3
#1 0x0000000060050c34 in _umtx_wait_uint_private (addr=<optimized out>, target_val=<optimized out>, tsz=<optimized out>, t=<optimized out>, where=<optimized out>)
at /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/bsd-user/freebsd/os-thread.c:258
#2 freebsd_lock_umutex (target_addr=4139717232, id=102353, ts=0x0, mode=<optimized out>) at /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/bsd-user/freebsd/os-thread.c:890
#3 0x000000006004a768 in do_freebsd__umtx_op (obj=4139717232, op=<optimized out>, val=0, uaddr=0, target_time=0)
at /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/bsd-user/freebsd/os-thread.h:359
#4 0x0000000060041405 in do_freebsd_syscall (cpu_env=0x860ca3198, num=454, arg1=<optimized out>, arg2=<optimized out>, arg3=<optimized out>, arg4=0, arg5=0, arg6=-167772160, arg7=102353, arg8=0)
at /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/bsd-user/syscall.c:1364
#5 0x0000000060038bb3 in target_cpu_loop (env=0x860ca3198) at /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/bsd-user/arm/target_arch_cpu.h:207
#6 0x0000000060038559 in cpu_loop (env=0xf6bf128c) at /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/bsd-user/main.c:121
#7 0x00000000600503ea in new_freebsd_thread_start (arg=<optimized out>) at /wrkdirs/usr/ports/emulators/qemu-user-static/work/qemu-bsd-user-4ef7d07/bsd-user/freebsd/os-thread.c:152
#8 0x00000000601ad1c6 in thread_start (curthread=0x8604d5200) at /usr/src/lib/libthr/thread/thr_create.c:292
#9 0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x7fffdfdfc000
PID USERNAME THR PRI NICE SIZE RES SWAP STATE C TIME WCPU COMMAND
64885 root 7 20 0 79M 12M 0 uwait 2 0:00 0.00% /usr/local/bin/qemu-arm-static ctfmerge -L VERSION -g -o bhnd.kld bhnd.o bhnd_subr.o bhnd_bus_if.o bhnd_erom.o bhnd_erom_if
64846 root 1 52 0 10M 1768K 0 wait 22 0:00 0.00% sh -e
46319 root 1 20 0 11M 2420K 0 select 8 0:00 0.00% /nxb-bin/usr/bin/make all DIRPRFX=bhnd/
Again: not repeatable for when/where --but has a thread
fanout involved.
Note: I had set things up to use the nxb-bin/... instead of
emulated compiling/linking/etc.
Another point: This was a native FreeBSD boot, not
under Hyper-V. But it is the same system and same
storage media as when used under Hyper-V.
===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)
More information about the freebsd-ports
mailing list