missing DTrace FBT return probes

[Bruce Evans]

On Thu, 13 Nov 2014, Navdeep Parhar wrote:

> On 11/13/14 08:33, Grier, James wrote:
>> On Wed, Jun 5, 2013 at 5:50 PM, Navdeep Parhar <np at freebsd.org> wrote: > 
>> A large number of kernel functions have an FBT entry probe but no return
>>> probe. I believe this is due to tail call optimization by the compiler.
>>> Should we disable this optimization for kernel configs that have DTrace
>>> support? The missing return probes make it very difficult to write
>>> DTrace scripts that want to set flags etc. at function entry and then
>>> clean them up on return.
>>> 
>>> A quick sample from a recent HEAD shows ~4000 out of ~27000 functions
>>> are missing return probes. See the list of functions in these files
>>> (the ones listed in entry-only.txt do not have return probes).
>>> 
>>> http://people.freebsd.org/~np/entry-only.txt
>>> http://people.freebsd.org/~np/entry.txt
>>> http://people.freebsd.org/~np/return.txt
>> 
>> By trial and error, I discovered that it’s ‘-funit-at-a-time’ that 
>> does the dirty work, so
>> using ‘-fno-unit-at-a-time’ with -O2 we get all the return fbt probes. 
>> I haven’t determined
>> the performance impact of this.
>
> Hmm, I thought it was -fno-optimize-sibling-calls that controlled this 
> particular optimization.

Perhaps it is all of these, plus -fno-inline-functions-called-once.
Inlining functions when not requested to do so gives undebuggable
code.  -funit-at-a-time exposes even static functions that are defined
after they are used to inlining.

> These days I compile my debug kernels with -O0.  There is a very significant 
> performance impact but it makes kgdb and DTrace very happy.

But -O0 means that you are not testing what will be used on production
systems.  gcc used to promise that -g not disturb anything and that
-O not make debugging impossible.  Lots of flags that were under
-O2 in gcc-3 crept into -O in gcc-4.  clang is worse -- its -O is more
agressive than gcc-4's -O2, and its -O2 is little different from its -O.
Apart from kgdb and DTrace, this breaks:
- profiling
- kernel stack traces from ddb
- ddb generally.  With no debugging symbol support, it is even harder to
   find where args and local variables are if they are optimized away.
   (In userland, they often cannot be found even with debugging symbol
   support.)  ddb's heuristic for finding args never worked for amd64
   and is now under #if !1 with a comment that dwarf2 is needed.  Thus,
   stack traces never showed any args correctly on amd64.  I recently
   noticed a case where gcc without many optimizations on i386 messed
   up the arg printing because the function was static and gcc
   "optimized" it by passing an arg in %eax instead of on the stack.

Bruce