Fun with gdb and threads...

[Peter Edwards]

Hi.
This might be of interest to anyone who has tried debugging
multi-threaded programs (of the libc_r variety) with gdb. This has been
bugging me for months, and I finally got frustrated enough to find out
what was going on.

The symptom:

Once you call any function that puts a thread to sleep, the target
process crashes (simple program, 1.c attached, and log of gdb killing it
in crash.txt)

The problem:

I traced this to an interaction between gdb and the threads scheduler.
The initial crash comes from gdb adding internal breakpoints in the
"(_)?(sig)?longjmp" functions. This breakpoint gets hit when the thread
scheduler calls "_thread_kern_sched"

After handling the breakpoint, gdb then needs to reset the instruction
pointer in the "current thread" to re-run the instruction the breakpoint
was at. However, at that point, gdb's freebsd_uthread_store_registers()
barfs, thinking that the thread in question is not "active", because its
not in state PS_RUNNING (it's just about to go to sleep). As a result,
it mucks up the resetting of the instruction pointer, because it thinks
it just needs to twiddle with the threads context, rather than the
"live" registers.

Once the process is resumed, it starts in the middle of whatever
instruction the breakpoint overwrote, and generally fscking things up.

The fix:

I added a couple of "nop"s to  "___longjmp", and created a new
entrypoint below them called "___longjmp_raw". This provides a way for
the libc_r library to avoid hitting the gdb breakpoints at sensitive
moments. All other consumers still work the exact same way (modulo the
time spent executing a couple of nops). The patch is attached, and makes
gdb behave perfectly for me.

Does anyone have any comments on this, or ideas on how to improve on it?
The only penalty I can see is an extra "nop" instruction for normal
longjmps, which I'll gladly trade for a usable debugger.

PS:

before anyone suggests it, I initially tried changing freebsd_uthread.c
to check for the active thread more effectively, as is done in
freebsd_uthread_fetch_registers, by comparing it with "_pthread_run",
rather than checking the state.

This improved things, but gdb still got confused, and started stopping
unexpectedly when it lost it's breakpoints, etc, so I figured the other
approach was probably going to be more stable.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: patch.txt
Type: text/x-patch
Size: 1851 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-current/attachments/20030807/1060c77d/patch.bin
-------------- next part --------------
petere at rocklobster$ gcc -o 1 -g -Wall -pthread 1.c
petere at rocklobster$ gdb ./1    
GNU gdb 5.2.1 (FreeBSD)
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-undermydesk-freebsd"...
(gdb) b threadFunc
Breakpoint 1 at 0x804861e: file 1.c, line 10.
(gdb) run
Starting program: /local/petere/1 

Breakpoint 1, threadFunc (arg=0x0) at 1.c:10
10          sleep(1);
(gdb) n

Program received signal SIGSEGV, Segmentation fault.
0x280d0138 in _longjmp () from /usr/lib/libc.so.5
(gdb)