DTrace userland support doesn't work on post r219107 FreeBSD 8.x

[Akbar Saidov]

Hi Guys,

George Neville-Neil suggested me to email this list describing the issue.

I am currently working on backporting DTrace userland support to
FreeBSD 8.2-Release
from FreeBSD 9-Stable in an effort to evaluate the performance of our
product, and perform an analysis and tracing of internal execution of our
userland process. Some time ago I learned from Rui Paulo that this work was
already done in FreeBSD 8.3 (r219107). To experiment, I installed FreeBSD
8.3-Release amd64, and recompiled the kernel and rebuilt the world per
instructions provided here https://wiki.freebsd.org/Dtrace/

However, running the basic test in
https://wiki.freebsd.org/DTrace/userlandfails in attempt to probe libc
(or
any system library) functions. (Note that the exact same issue happens when
I manually backport userland support to FreeBSD 8.2-Release)

# dtrace -s pid.d -c ./test
test: failed to compile script pid.d: line 4: probe description
pid2104:libc*:sleep:entry does not match any probes
User signal 1

Similarly, probing stops and an application gets killed when its time to probe
calls from ld-elf.so.1, libutil.so.8, libc.so.7, etc.

Happens when you run:

# dtrace -ln 'pid$target:::entry' -c date

   ID   PROVIDER            MODULE                          FUNCTION NAME
38195    pid2186              date                             usage entry
38196    pid2186              date                         badformat entry
38197    pid2186              date                       daysinmonth entry
38198    pid2186              date                             trans entry
38199    pid2186              date                          domktime entry
38200    pid2186              date                           adjhour entry
38201    pid2186              date                           adjyear entry
38202    pid2186              date                            adjmon entry
38203    pid2186              date                            adjday entry
38204    pid2186              date                            adjmin entry
38205    pid2186              date                           adjwday entry
38206    pid2186              date                      vary_destroy entry
38207    pid2186              date                       vary_append entry
38208    pid2186              date                        vary_apply entry
38209    pid2186              date                        netsettime entry
38210    pid2186              date                            _start entry
38211    pid2186              date                              main entry
User signal 1

And, what is more strange, probing returns fails as well:

# env DTRACE_DEBUG=1 dtrace -ln 'pid$target:::return' -c date

Truncated debug output:
...
libdtrace DEBUG: grabbed pid 2164 (cached)
libdtrace DEBUG: date stret 0 0 0 0
libdtrace DEBUG: st_size of __do_global_dtors_aux is zero
libdtrace DEBUG: st_size of frame_dummy is zero
libdtrace DEBUG: st_size of __do_global_ctors_aux is zero
libdtrace DEBUG: creating probe pid2164:date:usage:return
libdtrace DEBUG: mr sparkle: Pread() failed
libdtrace DEBUG: creating probe pid2164:date:badformat:return
libdtrace DEBUG: mr sparkle: Pread() failed
...
libdtrace DEBUG: creating probe pid2164:date:netsettime:return
libdtrace DEBUG: mr sparkle: Pread() failed
libdtrace DEBUG: creating probe pid2164:date:_start:return
libdtrace DEBUG: mr sparkle: Pread() failed
libdtrace DEBUG: creating probe pid2164:date:main:return
libdtrace DEBUG: mr sparkle: Pread() failed
libdtrace DEBUG: st_size of _fini is zero
libdtrace DEBUG: st_size of _init is zero
libdtrace DEBUG: st_size of _fini is zero
libdtrace DEBUG: set context to pid2164:::return [0] prp=0x0 attr=[u/u/c]
argc=0
libdtrace DEBUG: set label to <action list>
libdtrace DEBUG: set label to <NULL>
   ID   PROVIDER            MODULE                          FUNCTION NAME
date: failed to match pid2164:::return: No probe matches description
libdtrace DEBUG: killing pid 2164
User signal 1

Based on the email correspondence with Rui, he suspects that perhaps MFC
wasn't done correctly, but currently doesn't have any idea of what is going
on. I even tried FreeBSD 8-STABLE, and still same issue.

I was hoping someone could shed some light and point me to a direction of
how I can resolve this issue. I am open to any specific debugging
suggestions you may have, and I will be glad to try them out to get you
guys more information. I would very much appreciate any pointers.

Thanks,
-Akbar