How can a program destroy its thread local storage in FreeBSD 15?

From: Marcin Cieslak <saper_at_saper.info>
Date: Sat, 13 Sep 2025 10:44:28 UTC
I have a weird problem.

The program I am trying to run [1] worked last time
on  Mon Jun 17 23:24:58 2024 more or less on a then-current FREEBSD-CURRENT version.

Today I am running FreeBSD 15-CURRENT from 2 Aug 2025 (ac641d55ea0622f06baa60b9d22ef4880007d8c4).
This is all amd64. The code is 32-bit (cc -m32).

Since some time or more (few months at least) it started crashing
randomly - not always in the same place/backtrace but
always related to the use of the thread local storage (TLS).

By random crashes I mean that sometimes it completes one task
and breaks during the next run, but the pattern of the crashes
is pretty recognizable.

Sample backtrace:

(gdb) bt
#0  kill () at kill.S:4
#1  0x0040740c in exit_handler (sig=10) at ../../../forth/wrapper/wrapper.c:1264
#2  <signal handler called>
#3  0x205ae85b in tsd_get (init=false)
     at /usr/src/contrib/jemalloc/include/jemalloc/internal/tsd_malloc_thread_cleanup.h:52
#4  free_fastpath (ptr=ptr@entry=0x20662000, size=<optimized out>, size_hint=<optimized out>)
     at jemalloc_jemalloc.c:2792
#5  0x205ae82b in __free (ptr=0x20662000) at jemalloc_jemalloc.c:2851
#6  0x00407e4b in m_free (size=<optimized out>,
     adr=0x20662000 "CHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND\n\\ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE\n\\ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION\n\\ O"...) at ../../../forth/wrapper/wrapper.c:2131
#7  0x0040682a in fsyscall (callno=128, args=0x2068014c)
     at ../../../forth/wrapper/wrapper.c:592
#8  0x206800ab in ?? ()


Disassembling the free_fastpath function gives more clue
(gdb) frame 4
#4  free_fastpath (ptr=ptr@entry=0x20662000, size=<optimized out>, size_hint=<optimized out>)
     at jemalloc_jemalloc.c:2792
2792		tsd_t *tsd = tsd_get(false);
(gdb) disass
Dump of assembler code for function free_fastpath:
    0x205ae840 <+0>:	push   %ebp
    0x205ae841 <+1>:	mov    %esp,%ebp
    0x205ae843 <+3>:	push   %ebx
    0x205ae844 <+4>:	push   %edi
    0x205ae845 <+5>:	push   %esi
    0x205ae846 <+6>:	sub    $0x8,%esp
    0x205ae849 <+9>:	call   0x205ae84e <free_fastpath+14>
    0x205ae84e <+14>:	pop    %ebx
    0x205ae84f <+15>:	add    $0x62d96,%ebx
    0x205ae855 <+21>:	mov    -0x3d58(%ebx),%edx
=> 0x205ae85b <+27>:	mov    %gs:0x0,%eax
(gdb) info r
eax            0x80                128
ecx            0x20662000          543563776
edx            0xffffee30          -4560
ebx            0x206115e4          543233508
esp            0xffffd4cc          0xffffd4cc
ebp            0xffffd4e0          0xffffd4e0
esi            0x20662000          543563776
edi            0x20670020          543621152
eip            0x205ae85b          0x205ae85b <free_fastpath+27>
eflags         0x210216            [ PF AF IF RF ID ]
cs             0x33                51
ss             0x3b                59
ds             0x3b                59
es             0x7798              30616
fs             0x13                19
gs             0x1b                27
fs_base        0x0                 0
gs_base        0x0                 0

The crashes always occur with the attempt to fetch the TLS
(mov %gs:0x0, %eax), it happens in malloc(), free()
as well as in the printf() family of functions, since TLS
is apparently being used there to store locale information (ugh).

This code did not change. It does not use threads.

I wonder why it broke.

What kind of helped to fix the memory allocator  was to replace malloc()
and free() from jemalloc with the simple implementation from
our rtld-elf, which is not useing TLS. But I am not going
to provide a TLS-free printf(3) family of functions.

If anyone has some hints how to troubleshoot that kind of problems,
I would be very grateful. I wonder also why it worked more than a year ago.

Thank you,
Marcin

[1] https://github.com/MitchBradley/openfirmware with some minimal changes
to run it on FreeBSD