svn commit: r324938 - head/contrib/jemalloc/include/jemalloc/internal

[Michal Meloun]

On 27.10.2017 17:21, Brooks Davis wrote:
> On Fri, Oct 27, 2017 at 06:08:41PM +0300, Konstantin Belousov
> wrote:
>> On Fri, Oct 27, 2017 at 02:53:26PM +0200, Michal Meloun wrote:
>>> Sorry for top posting That's pity, we have clear problem in
>>> rtld code :( See: 
>>> ----------------------------------------------------- RESCUE
>>> WITHOUT JEMALLOC_ALIGNED(16); 
>>> ----------------------------------------------------- Program
>>> Headers: TLS            0xa732b0 0x00a832b0 0x00a832b0 0x00b40
>>> 0x011bc R   0x8 Section Headers: 04     .tdata .tbss
>>> .init_array .fini_array .jcr .got Dump: 00a832b0
>>> <__je_tsd_tls+0xa832b0>: a832b0:       00000005
>>> 
>>> GDB (gdb) b tsd_fetch_impl Breakpoint 1 at 0x7c4c08:
>>> tsd_fetch_impl. (6 locations) (gdb) r Starting program:
>>> /usr/src/rescue.noalign sh
>>> 
>>> Breakpoint 1, tsd_fetch_impl (init=true, minimal=false) at 
>>> /usr/src/contrib/jemalloc/include/jemalloc/internal/tsd.h:261 
>>> 261             tsd_t *tsd = tsd_get(init); (gdb) n 263
>>> if (!init && tsd_get_allocates() && tsd == NULL) {
>>> 
>>> (gdb) p tsd $1 = (tsd_t *) 0x20c83008
>>> 
>>> (gdb) p *tsd $2 = {state = 5 '\005', ....
>>> 
>>> (gdb) p *((tsd_t *)0x00a832b0) $3 = {state = 5 '\005', ...
>>> 
>>> 
>>> 
>>> ----------------------------------------------------- RESCUE
>>> WITH JEMALLOC_ALIGNED(16); 
>>> ----------------------------------------------------- Program
>>> Headers: TLS            0xa732b0 0x00a832b0 0x00a832b0 0x00b40
>>> 0x011bc R   0x10 Section Headers: 04     .tdata .tbss
>>> .init_array .fini_array .jcr .got Dump: 00a832b0
>>> <__je_tsd_tls+0xa832b0>: a832b0:       00000005
>>> 
>>> GDB (gdb) b tsd_fetch_impl Breakpoint 1 at 0x7c4c08:
>>> tsd_fetch_impl. (6 locations) (gdb) r Starting program:
>>> /usr/obj/usr/src/rescue/rescue/rescue sh Breakpoint 1,
>>> tsd_fetch_impl (init=true, minimal=false) at 
>>> /usr/src/contrib/jemalloc/include/jemalloc/internal/tsd.h:261 
>>> 261             tsd_t *tsd = tsd_get(init); (gdb) n 263
>>> if (!init && tsd_get_allocates() && tsd == NULL) {
>>> 
>>> (gdb) p tsd $1 = (tsd_t *) 0x20c83010
>>> 
>>> (gdb) p *tsd $2 = {state = 0 '\000', ...
>>> 
>>> (gdb) p *((tsd_t *)0x00a832b0) $3 = {state = 5 '\005', ...
>>> 
>>> !!!! BUT p *(tsd - 8 bytes) !!!!!!!!!! (gdb) p *((tsd_t
>>> *)0x20c83008) $4 = {state = 5 '\005', ...
>>> 
>>> ----------------------------------------------------- So it's
>>> clear that:
>>> 
>>> - both binaries are valid, .tdata section have valid data. -
>>> requested alignment is propagated to binary. - .tdata section
>>> is properly loaded to memory because p *((tsd_t *)0x00a832b0)
>>> is {state = 5 '\005' in both cases
>>> 
>>> - a per thread copy of .tdata respect requested alignment but
>>> the original data was copied to  unaligned address. because for
>>> aligned binary p *tsd is {state = 0 '\000', ... p *(tsd - 8
>>> bytes) is {state = 5 '\005'
>>> 
>>> I'm right? Kib, please, can you help us?
>> 
>> Does it happen for rescue binary ?
>> 
>> Note that the binary is linked static, so the problem is in
>> lib/libc/gen/tls.c and not in rtld.  There, I do not see any real
>> use of the phdr' p_align value.
>> 
Ahh, right, good catch.
>> BTW, is rescue linked to libthr ?
Imho not.
> 
> There isn't alignment support for TLS in static binaries.  I've
> fixed this in CheriBSD and am planning to upstream the fixes at
> some point. The fix for variant I is in:
> 
> https://github.com/CTSRD-CHERI/cheribsd/commit/3cfb124ebb9fdb545dad8436a04dd58c05b33f4b
>
> 
The real alignment have no effect for program itself, it's important
only for AddressSanitizer.

But there is something what's still missing me.
The rescue binary is statically linked, so whole program have only one
TLS section and full TLS layout is determined by linker.
The problematic structure,  tsd_tls, is first one in TLS section group:

readelf -a /usr/obj/usr/src/rescue/rescue/rescue | grep TLS
518961: 0000000000000000  2880 TLS     GLOBAL DEFAULT    9 __je_tsd_tls
and

objdump -d -j .tdata /usr/obj/usr/src/rescue/rescue/rescue
00a832b0 <__je_tsd_tls+0xa832b0>:
a832b0:       00000005

but disassembled code expect it at TLS ptr + 0x8 (for code without
JEMALLOC_ALIGNED) or at TLS ptr + 0x10 (for code with JEMALLOC_ALIGNED)

[without JEMALLOC_ALIGNED]
  7c4d90:       e59f0014        ldr     r0, [pc, #20]   ; 7c4dac
<tsd_get+0x80>
  7c4d94:       e58d0004        str     r0, [sp, #4]
  7c4d98:       eb031d14        bl      88c1f0 <__aeabi_read_tp>
  7c4d9c:       e59dc004        ldr     ip, [sp, #4]
  7c4da0:       e080000c        add     r0, r0, ip
  7c4da4:       e1a0d00b        mov     sp, fp
  7c4da8:       e8bd8800        pop     {fp, pc}
  7c4dac:       00000008        .word   0x00000008  [0x10 for code
with  JEMALLOC_ALIGNED]

so someone other allocates at least one byte from start of TLS and
linker is aware about this. But again, .data is first member of TLS
and tsd_tls is first of .tdata and the 0x5 is value of first member of
tsd_tls structure (.state = tsd_state_uninitialized).

So who consumes the first bytes in TLS ????

Michal