i386 on amd64 can fail to return from cond_wait_user, using basically 100% of a FreeBSD cpu

Reply: Konstantin Belousov : "Re: i386 on amd64 can fail to return from cond_wait_user, using basically 100% of a FreeBSD cpu"
Go to: [ bottom of page ] [ top of archives ] [ this month ]
From: Mark Millard <marklmi_at_yahoo.com>
Date: Sat, 05 Jul 2025 06:01:22 UTC
Some package builds are failing on the port-packages build cluster
machines that do i386 builds during the following code. The analysis
is from replication in a personal context (using poudriere bulk with
-i), using poudriere-devel instead. I'll note that the personal
context is from using PkgBase 14.3-RELEASE in the poudriere jail.
I also installed most of the realted *-dbg* PkgBase packages in order
to get the nicer backtracing.

(gdb) bt
#0  _umtx_op_err () at /home/pkgbuild/worktrees/releng/14.3/lib/libthr/arch/i386/i386/_umtx_op_err.S:37
#1  0x2499f897 in _thr_umtx_timedwait_uint (mtx=0x249a365c, id=0, clockid=4, abstime=0x0, shared=0) at /home/pkgbuild/worktrees/releng/14.3/lib/libthr/thread/thr_umtx.c:233
#2  0x24995b26 in _thr_sleep (curthread=0x24d36004, clockid=4, abstime=0x0) at /home/pkgbuild/worktrees/releng/14.3/lib/libthr/thread/thr_kern.c:197
#3  0x24990beb in cond_wait_user (cvp=0x24dfa8a0, mp=0x24d38d04, abstime=<optimized out>, cancel=<optimized out>) at /home/pkgbuild/worktrees/releng/14.3/lib/libthr/thread/thr_cond.c:317

NOTE: cond_wait_user never returns but #2..#0 repeat (observed by
repeated ^c and bt usage).

(i386 is the oddball with 32-bit time_t but I do not know
if that is involved here.)

#4  cond_wait_common (cond=<optimized out>, mutex=<optimized out>, abstime=0x0, cancel=1) at /home/pkgbuild/worktrees/releng/14.3/lib/libthr/thread/thr_cond.c:377
#5  0x24990e8f in __thr_cond_wait (cond=0x23b9b4f4, mutex=0x23b9b4ec) at /home/pkgbuild/worktrees/releng/14.3/lib/libthr/thread/thr_cond.c:392
#6  0x23be1e4b in uv_cond_wait () from /usr/local/lib/libuv.so.1
#7  0x024bd497 in node::NodePlatform::DrainTasks(v8::Isolate*) ()
#8  0x0232f5b6 in node::SpinEventLoopInternal(node::Environment*) ()
#9  0x02485bf0 in node::NodeMainInstance::Run() ()
#10 0x023eaba1 in node::Start(int, char**) ()
#11 0x24a1da85 in __libc_start1 (argc=5, argv=0xffffda3c, env=0xffffda54, cleanup=0x23b73020 <rtld_nop_exit>, mainX=0x314a720 <main>)
    at /home/pkgbuild/worktrees/releng/14.3/lib/libc/csu/libc_start1.c:157
#12 0x0232d0a8 in _start ()

www/librewolf and other firefox related package builds can do
this until a 7200 sec timeout by poudriere occurs:

=>> Killing runaway build after 7200 seconds with no output

I'll note that truss did not generate any output when used to
watch the process that was stuck. It appears to be a world-internal
problem.


For reference (note the thread with "35:49.90" that was the
one stuck looping in such a manor):

# ps -Halxdww
. . .
  0  4221 93681  9  15  0  16868   5432 wait     I     0     0:00.00 | |         `-- sh: poudriere[release-i386-alt][20]: build_pkg (librewolf-139.0.4) (sh)
  0  4222  4221 26  59  0   6312   3332 wait     IsJ   -     0:00.01 | |           `-- /usr/bin/make -C /usr/ports/www/librewolf build
  0  4246  4222 17  59  0   5064   2740 wait     IJ    -     0:00.00 | |             `-- /bin/sh -e -c (cd /wrkdirs/usr/ports/www/librewolf/work/.build; if ! /usr/bin/env -i HOME=/wrkdirs/usr/ports/www/librewolf/work  MACHINE_ARCH=i386  PWD="${PWD}"  GIT_CEILING_DIRECTORIES=/wrkdirs/usr/ports/www/librewolf/work  __MAKE_CONF=/nonexistent OSVERSION=1403000 PATH=/wrkdirs/usr/ports/www/librewolf/work/.bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin TMPDIR=/tmp UNAME_m=i386 UNAME_p=i386 UNAME_r=14.3-RELEASE UNAME_v=FreeBSD\\ 14.3-RELEASE LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 XDG_DATA_HOME=/wrkdirs/usr/ports/www/librewolf/work  XDG_CONFIG_HOME=/wrkdirs/usr/ports/www/librewolf/work  XDG_CACHE_HOME=/wrkdirs/usr/ports/www/librewolf/work/.cache  HOME=/wrkdirs/usr/ports/www/librewolf/work TMPDIR="/tmp" PATH=/wrkdirs/usr/ports/www/librewolf/work/.bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin PKG_CONFIG_LIBDIR=/wrkdirs/usr/ports/www/librewolf/work/.pkgconfig:/usr/local/libdata/pkgconfig:/usr/local/share/pkgconfig:/usr/libdata/pkgconfig MK_DEBUG_FILES=no MK_KERNEL_SYMBOLS=no SHELL=/bin/sh NO_LINT=YES PREFIX=/usr/local  LOCALBASE=/usr/local  CC="/usr/local/bin/clang19" CFLAGS="-O2 -pipe  -O3 -DLIBICONV_PLUG -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing "  CPP="/usr/local/bin/clang-cpp19" CPPFLAGS="-DLIBICONV_PLUG -isystem /usr/local/include"  LDFLAGS=" -Wl,--as-needed -Wl,--undefined-version -fstack-protector-strong " LIBS="-L/usr/local/lib"  CXX="/usr/local/bin/clang++19" CXXFLAGS="-O2 -pipe -O3 -DLIBICONV_PLUG -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing  -DLIBICONV_PLUG -isystem /usr/local/include " BSD_INSTALL_PROGRAM="install   -m 555"  BSD_INSTALL_LIB="install   -m 0644"  BSD_INSTALL_SCRIPT="install  -m 555"  BSD_INSTALL_DATA="install  -m 0644"  BSD_INSTALL_MAN="install  -m 444" /usr/local/bin/gmake -f  Makefile   all; then  if [ -n "" ] ; then  echo "===> Compilation failed unexpectedly.";  (echo "") | /usr/bin/fmt 75 79 ;  fi;  false;  fi)
  0  4247  4246 30  29  0   7580   4980 wait     IJ    -     0:00.01 | |               `-- /usr/local/bin/gmake -f Makefile all
  0 57541  4247 19  37  0   8020   5472 wait     IJ    -     0:00.03 | |                 `-- /usr/local/bin/gmake misc
  0 57542 57541 29  59  0   8020   5480 wait     IJ    -     0:00.03 | |                   `-- /usr/local/bin/gmake recurse_misc
  0 57756 57542  3  59  0  27412  24160 select   IJ    -     0:00.07 | |                     `-- /wrkdirs/usr/ports/www/librewolf/work/.mozbuild/srcdirs/librewolf-139.0.4-1-f0fbc434e1bd/_virtualenvs/build/bin/python -m mozbuild.action.file_generate /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/python/mozbuild/mozbuild/action/node.py generate devtools/client/debugger/src/actions/ast/node.stub devtools/client/debugger/src/actions/ast/.deps/node.stub.pp devtools/client/debugger/src/actions/ast/.deps/node.stub.stub /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/shared/build/build.js /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/debugger/src/actions/ast/index.js /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/debugger/src/actions/ast/setInScopeLines.js /wrkdirs/usr/ports/www/librewolf/work/.build//dist/bin/browser/chrome/devtools/modules/devtools/client/debugger/src/actions/ast (python3.11)
  0 57757 57756 19  59  0 147928 131892 uwait    IJ    -     0:00.09 | |                       |-- /usr/local/bin/node /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/shared/build/build.js /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/debugger/src/actions/ast/index.js /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/debugger/src/actions/ast/setInScopeLines.js /wrkdirs/usr/ports/www/librewolf/work/.build//dist/bin/browser/chrome/devtools/modules/devtools/client/debugger/src/actions/ast
  0 57757 57756  2  59  0 147928 131892 kqread   IJ    -     0:00.00 | |                       |-- /usr/local/bin/node /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/shared/build/build.js /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/debugger/src/actions/ast/index.js /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/debugger/src/actions/ast/setInScopeLines.js /wrkdirs/usr/ports/www/librewolf/work/.build//dist/bin/browser/chrome/devtools/modules/devtools/client/debugger/src/actions/ast
  0 57757 57756  3  59  0 147928 131892 uwait    IJ    -     0:00.00 | |                       |-- /usr/local/bin/node /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/shared/build/build.js /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/debugger/src/actions/ast/index.js /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/debugger/src/actions/ast/setInScopeLines.js /wrkdirs/usr/ports/www/librewolf/work/.build//dist/bin/browser/chrome/devtools/modules/devtools/client/debugger/src/actions/ast
  0 57757 57756 21  59  0 147928 131892 uwait    IJ    -     0:00.00 | |                       |-- /usr/local/bin/node /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/shared/build/build.js /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/debugger/src/actions/ast/index.js /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/debugger/src/actions/ast/setInScopeLines.js /wrkdirs/usr/ports/www/librewolf/work/.build//dist/bin/browser/chrome/devtools/modules/devtools/client/debugger/src/actions/ast
  0 57757 57756 25 141  0 147928 131892 -        RJ    -    35:49.90 | |                       |-- /usr/local/bin/node /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/shared/build/build.js /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/debugger/src/actions/ast/index.js /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/debugger/src/actions/ast/setInScopeLines.js /wrkdirs/usr/ports/www/librewolf/work/.build//dist/bin/browser/chrome/devtools/modules/devtools/client/debugger/src/actions/ast
  0 57757 57756 30  59  0 147928 131892 uwait    IJ    -     0:00.00 | |                       |-- /usr/local/bin/node /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/shared/build/build.js /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/debugger/src/actions/ast/index.js /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/debugger/src/actions/ast/setInScopeLines.js /wrkdirs/usr/ports/www/librewolf/work/.build//dist/bin/browser/chrome/devtools/modules/devtools/client/debugger/src/actions/ast
  0 57757 57756 12  59  0 147928 131892 usem     IJ    -     0:00.00 | |                       `-- /usr/local/bin/node /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/shared/build/build.js /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/debugger/src/actions/ast/index.js /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/debugger/src/actions/ast/setInScopeLines.js /wrkdirs/usr/ports/www/librewolf/work/.build//dist/bin/browser/chrome/devtools/modules/devtools/client/debugger/src/actions/ast
. . .


# poudriere jail -jrelease-i386 -i
Jail name:         release-i386
Jail version:      14.3-RELEASE
Jail arch:         i386
Jail method:       pkgbase
Jail mount:        /usr/local/poudriere/jails/release-i386
Jail fs:           zoptb/poudriere/jails/release-i386
Jail updated:      2025-06-28 21:53:52
Jail pkgbase:      disabled

# uname -apKU
FreeBSD 7950X3D-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT main-n278320-3a33e39edd48 GENERIC-NODEBUG amd64 amd64 1500048 1500048

# ~/fbsd-based-on-what-commit.sh -C /usr/ports-alt/
6a6a0f11043c (HEAD -> main, freebsd/main, freebsd/HEAD) graphics/kdegraphics: install colord-kde
Author:     Max Brazhnikov <makc@FreeBSD.org>
Commit:     Max Brazhnikov <makc@FreeBSD.org>
CommitDate: 2025-06-28 18:03:24 +0000
branch: main
merge-base: 6a6a0f11043cea55e789395d2f89b091c1cdf97c
merge-base: CommitDate: 2025-06-28 18:03:24 +0000
n711132 (--first-parent --count for merge-base)


https://portsfallout.com/fallout?port=%2Flibrewolf&maintainer=&env=&category=build%2Frunaway&flavor=

shows as having such failures for www/librewolf:

135i386-default
main-i386-default
142i386-quarterly
135i386-quarterly
134i386-default
134i386-quarterly

Interestingly, 142i386-default is not showing up there, But:

https://pkg-status.freebsd.org/beefy21/data/142i386-default/557ad2fd0e8a/logs/errors/librewolf-140.0.2.log

is an example.

(Looks like 142i386-default might not be getting fallout reports?)

===
Mark Millard
marklmi at yahoo.com