malloc: bug or feature ?

[James R. Van Artsalen]

Peter Wemm wrote:

>The SEGV looks suspicious though.  I wonder if malloc is trying to put 
>the junk in without checking if malloc failed.
>
>Anyway, try "env MALLOC_OPTIONS=aj ./a.out"  See 'man malloc.conf' etc 
>for more info.
>  
>

I'm too sleepy to do a good analysis, but a glance I think there's a 
problem here.  Consider the brk() code below.  There's one return 
address then one register on the stack, balanced stack, no problem.  But 
follow the error path if the SYSCALL returns an error:

0000000000409138 <_brk>:
  409138:       57                      push   %rdi
  409139:       eb 15                   jmp    409150 <ok>
  40913b:       90                      nop   

000000000040913c <brk>:
  40913c:       57                      push   %rdi
  40913d:       48 89 f8                mov    %rdi,%rax
  409140:       48 39 05 71 71 10 00    cmp    %rax,1077617(%rip)        
# 5102b8 <.minbrk>
  409147:       76 07                   jbe    409150 <ok>
  409149:       48 8b 3d 68 71 10 00    mov    1077608(%rip),%rdi        
# 5102b8 <.minbrk>

0000000000409150 <ok>:
  409150:       48 c7 c0 11 00 00 00    mov    $0x11,%rax
  409157:       49 89 ca                mov    %rcx,%r10
  40915a:       0f 05                   syscall
  40915c:       72 15                   jb     409173 <err>
  40915e:       48 8b 44 24 00          mov    0x0(%rsp,1),%rax
  409163:       48 89 05 56 71 10 00    mov    %rax,1077590(%rip)        
# 5102c0 <.curbrk>
  40916a:       48 c7 c0 00 00 00 00    mov    $0x0,%rax
  409171:       5f                      pop    %rdi
  409172:       c3                      retq  

0000000000409173 <err>:
  409173:       e9 ac 03 00 00          jmpq   409524 <.cerror>

At this point there is still one reg and one return addr on the stack.

What I think goes wrong is that .cerror assumes the top-of-stack is a 
return address - it just does a RETQ.  But we still have %rdi on the 
stack from brk().  So $pc wanders off into the weeds when RETQ is done. 

brk() is guilty of going to .cerror without tidying up the stack first.

0000000000409524 <.cerror>:
  409524:       50                      push   %rax
  409525:       e8 16 00 00 00          callq  409540 <__error_unthreaded>
  40952a:       59                      pop    %rcx
  40952b:       48 89 08                mov    %rcx,(%rax)
  40952e:       48 c7 c0 ff ff ff ff    mov    $0xffffffffffffffff,%rax
  409535:       48 c7 c2 ff ff ff ff    mov    $0xffffffffffffffff,%rdx
  40953c:       c3                      retq  
  40953d:       90                      nop   
  40953e:       90                      nop   
  40953f:       90                      nop   

0000000000409540 <__error_unthreaded>:
  409540:       b8 a8 6a 51 00          mov    $0x516aa8,%eax
  409545:       c3                      retq