"unstable" sdt probes

[Mark Johnston]

On Fri, Oct 25, 2013 at 07:30:45PM +0300, Andriy Gapon wrote:
> on 24/10/2013 19:16 Mark Johnston said the following:
> > On Thu, Oct 24, 2013 at 01:20:17PM +0300, Andriy Gapon wrote:
> >>
> >> Can our SDT (for kernel) implementation support probes with unspecified types
> >> for arguments?
> >> I would like to get DTRACE_PROBE*() which can be found in the code with
> >> OpenSolaris origins (e.g. ZFS) to work.  With minimal efforts :-)
> > 
> > Hm, it looks like illumos uses the first argument to DTRACE_PROBE* to
> > specify both the provider name and probe name; if no provider name is
> > given (based on a lookup in a table in sdt_subr.c), a default "sdt"
> > provider is used. It looks like this is what's happening for the ZFS
> > probes.
> > 
> > I'd suggest something like the following:
> > - to kern_sdt.c, add
> > 
> > SDT_PROVIDER_DEFINE(sdt);
> > 
> > - add DTRACE_PROBE* macros to sdt.h which invoke SDT_PROBE* with the sdt
> >   provider, i.e. something like
> > 
> > #define	DTRACE_PROBE1(name, type, arg) \
> > 	SDT_PROBE1(sdt, , , name, arg)
> > 
> > - add a FreeBSD-only zfs_dtrace.c which contains the SDT_PROBE_DEFINE*
> >   invocations for the ZFS probes. You can define the types there, or not
> >   (using an empty string or NULL should work, I'm not sure).
> > 
> > This won't work for illumos code where the probes specify a provider,
> > but I think that's ok. I can do the first two steps if you agree with
> > this approach. I don't have any way to test the ZFS probes at the
> > moment, but I guess I can provide a zfs_dtrace.c too if you (or anyone
> > else) can test.
> 
> Mark,
> 
> thank you for the ideas!  The approach sounds fine to me.
> I plan to have some time to work on this next week.
> I will definitely be able to test things and maybe even develop something.
> So, thank you again.

The patch here is what I had in mind:
http://people.freebsd.org/~markj/patches/zfs_probes.diff

I've only compile-tested it, but it should create illumos-compatible ZFS
probes without changing any ZFS code, assuming I understand exactly how
they're creating/naming probes. :)

-Mark

> 
> BTW, I've been pondering an idea of reimplementing how the SDT probes get
> called.  In FreeBSD we have a special hook function pointer that we check for
> not being NULL and then make a function call.
> In illumos they compile the code with an unconditional function call.  This way
> the probe parameters are placed into the proper registers (or stack locations).
>  But during run-time linking the call instructions are replaced with series of
> 1-byte NOP instructions (5 x 0x90 for amd64).  When a probe gets activated then
> the first of those NOPs gets replaced with 0xf0 (lock prefix), which results in
> an invalid instruction (and that happens atomically).  So, that allows for the
> SDT hook to be invoked via the trap handler.
> 
> So, I think that that results in less overhead for inactive probes, but probably
> in more overhead for active probes.  There is a trade off, but I believe that
> less overhead for inactive probes is preferred.
> 
> -- 
> Andriy Gapon