How can a program destroy its thread local storage in FreeBSD 15?
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Sat, 13 Sep 2025 10:44:28 UTC
I have a weird problem.
The program I am trying to run [1] worked last time
on Mon Jun 17 23:24:58 2024 more or less on a then-current FREEBSD-CURRENT version.
Today I am running FreeBSD 15-CURRENT from 2 Aug 2025 (ac641d55ea0622f06baa60b9d22ef4880007d8c4).
This is all amd64. The code is 32-bit (cc -m32).
Since some time or more (few months at least) it started crashing
randomly - not always in the same place/backtrace but
always related to the use of the thread local storage (TLS).
By random crashes I mean that sometimes it completes one task
and breaks during the next run, but the pattern of the crashes
is pretty recognizable.
Sample backtrace:
(gdb) bt
#0 kill () at kill.S:4
#1 0x0040740c in exit_handler (sig=10) at ../../../forth/wrapper/wrapper.c:1264
#2 <signal handler called>
#3 0x205ae85b in tsd_get (init=false)
at /usr/src/contrib/jemalloc/include/jemalloc/internal/tsd_malloc_thread_cleanup.h:52
#4 free_fastpath (ptr=ptr@entry=0x20662000, size=<optimized out>, size_hint=<optimized out>)
at jemalloc_jemalloc.c:2792
#5 0x205ae82b in __free (ptr=0x20662000) at jemalloc_jemalloc.c:2851
#6 0x00407e4b in m_free (size=<optimized out>,
adr=0x20662000 "CHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND\n\\ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE\n\\ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION\n\\ O"...) at ../../../forth/wrapper/wrapper.c:2131
#7 0x0040682a in fsyscall (callno=128, args=0x2068014c)
at ../../../forth/wrapper/wrapper.c:592
#8 0x206800ab in ?? ()
Disassembling the free_fastpath function gives more clue
(gdb) frame 4
#4 free_fastpath (ptr=ptr@entry=0x20662000, size=<optimized out>, size_hint=<optimized out>)
at jemalloc_jemalloc.c:2792
2792 tsd_t *tsd = tsd_get(false);
(gdb) disass
Dump of assembler code for function free_fastpath:
0x205ae840 <+0>: push %ebp
0x205ae841 <+1>: mov %esp,%ebp
0x205ae843 <+3>: push %ebx
0x205ae844 <+4>: push %edi
0x205ae845 <+5>: push %esi
0x205ae846 <+6>: sub $0x8,%esp
0x205ae849 <+9>: call 0x205ae84e <free_fastpath+14>
0x205ae84e <+14>: pop %ebx
0x205ae84f <+15>: add $0x62d96,%ebx
0x205ae855 <+21>: mov -0x3d58(%ebx),%edx
=> 0x205ae85b <+27>: mov %gs:0x0,%eax
(gdb) info r
eax 0x80 128
ecx 0x20662000 543563776
edx 0xffffee30 -4560
ebx 0x206115e4 543233508
esp 0xffffd4cc 0xffffd4cc
ebp 0xffffd4e0 0xffffd4e0
esi 0x20662000 543563776
edi 0x20670020 543621152
eip 0x205ae85b 0x205ae85b <free_fastpath+27>
eflags 0x210216 [ PF AF IF RF ID ]
cs 0x33 51
ss 0x3b 59
ds 0x3b 59
es 0x7798 30616
fs 0x13 19
gs 0x1b 27
fs_base 0x0 0
gs_base 0x0 0
The crashes always occur with the attempt to fetch the TLS
(mov %gs:0x0, %eax), it happens in malloc(), free()
as well as in the printf() family of functions, since TLS
is apparently being used there to store locale information (ugh).
This code did not change. It does not use threads.
I wonder why it broke.
What kind of helped to fix the memory allocator was to replace malloc()
and free() from jemalloc with the simple implementation from
our rtld-elf, which is not useing TLS. But I am not going
to provide a TLS-free printf(3) family of functions.
If anyone has some hints how to troubleshoot that kind of problems,
I would be very grateful. I wonder also why it worked more than a year ago.
Thank you,
Marcin
[1] https://github.com/MitchBradley/openfirmware with some minimal changes
to run it on FreeBSD