zero-cost SDT probes

[Mark Johnston]

On Sat, Nov 21, 2015 at 10:29:37PM -0800, Simon J. Gerraty wrote:
> Mark Johnston <markj at freebsd.org> wrote:
> > For the past while I've been experimenting with various ways to
> > implement "zero-cost" SDT DTrace probes. Basically, at the moment an SDT
> > probe site expands to this:
> 
> Would it be feasible to compile the probes into the kernel
> as active calls to a registrar function?
> That would eliminate all the complexity of finding PC's
> though you'd probably need to pass extra args to convey the point of the
> probe?
> 
> It would hurt boot time a little too - each probe point would make a
> call to register itself (and get overwritten with nops as a reward) but
> very simple?

I considered such an approach but didn't pursue it for a few reasons:
- We'd have to pass a unique probe site identifier as an argument, which
  requires at least one extra instruction at the probe site.
- If the probe site is a tail call, how can the registrar find the
  correct caller?
- If a probe site isn't patched until multiple CPUs have started, how do
  we safely overwrite the call site in the face of the possibility that
  another thread is executing the call at the same time? When it comes
  to enabling or disabling a probe, we only need to write a single byte,
  but overwriting multiple bytes seems unsafe.

I think the last point could possibly be addressed by overwriting the
first byte of the call with a breakpoint before overwriting the rest of
the call site with NOPs, using the breakpoint handler to fix up any
threads that reached the probe site as it was being modified. But this
detracts a bit from the simplicity of the approach.