DTrace network providers

[Mark Johnston]

On Tue, Aug 20, 2013 at 11:44:56PM -0700, Brendan Gregg wrote:
> G'Day,
> 
> On Tue, Aug 20, 2013 at 10:00 PM, Mark Johnston <markj at freebsd.org> wrote:
> 
> > Hello!
> >
> > I've ported the ip, tcp and udp DTrace providers to FreeBSD, following
> > the Solaris documentation here:
> >
> > https://wikis.oracle.com/display/DTrace/ip+Provider
> > https://wikis.oracle.com/display/DTrace/tcp+Provider
> > https://wikis.oracle.com/display/DTrace/udp+Provider
> >
> >
> Fantastic!
> 
> 
> > My implementation of these providers makes use of dynamic translators,
> > for which FreeBSD support was added in r254468; this patch won't compile
> > with earlier revisions. The use of dynamic translators means that
> > existing DTrace scripts which use these providers will just work when run
> > on FreeBSD - no modifications needed. In particular, all of the examples
> > in the links above will work properly on FreeBSD with my diff.
> >
> > I've collected a bunch of example scripts for these providers and placed
> > them here:
> >
> > http://people.freebsd.org/~markj/dtrace/network-providers/
> >
> >
> They look familiar. :-) Which is a good sign for this port, as that's
> what's supposed to happen: the same scripts run on any OS.

Yep - by "collect" I mostly meant "copied from the DTrace book." :)
They were very useful for testing. At first all the examples had to be
changed to use the xlate operator on arguments, but now everything just
works.

> 
> To run one you just need to execute "dtrace -s <script>".
> >
> 
> Or chmod 755 the ones with interpreter lines.
> 
> 
> > In general these providers make it trivial to monitor or count packets
> > and bytes on a per-host/port/protocol/interface basis. One can also do
> > neat things like watch TCP connection state transitions in real time
> > (tcpstate.d) and measure connection latency. All of the probes
> > correspond to logical events in their respective protocol
> > implementations; all of the providers have send and receive probes, and
> > the tcp provider has a few more.
> >
> > I didn't have to make any major changes to add support for these
> > providers, but I've made a few small tweaks:
> >
> > 1. Add a little helper function for TCP state changes instead of
> >    sprinkling a bunch of SDT_PROBE calls throughout the code.
> >
> 
> Makes sense.
> 
> 
> > 2. Set the IPv6 payload size in TCP and UDP before calling ip6_output().
> >    This is done for the send probes so that the ipinfo_t argument
> >    can be used to get the payload size. It's not quite correct since it
> >    doesn't include the length of IPv6 option headers (they aren't known
> >    yet), but I think that's ok for the purposes of these providers.
> >
> 
> Excluding the IP header length is correct for ipinfo_t->ip_plength, which
> is just supposed to be the payload length.
> 
> I see the patch makes an assumption about IPv4 header size being 20 bytes,
> instead of picking it from the header (IHL), like:
> 
>         ip_plength = ntohs(I->ipha_length) -
>             ((I->ipha_version_and_hdr_length & 0xf) << 2);

Ah, thanks for pointing this out! I've fixed this in the revision here:

http://people.freebsd.org/~markj/patches/network-providers/network-providers-2.diff

Now that field is obtained with:

ip_plength = p == NULL ? 0 :
    ((struct ip *)p)->ip_v == 4 ?
    ntohs(((struct ip *)p)->ip_len) - (((struct ip *)p)->ip_hl << 2) :
    ntohs(((struct ip6_hdr *)p)->ip6_ctlun.ip6_un1.ip6_un1_plen);

> 
> 3. Set the IPv4 header version in udp_output(). This is already done for
> >    IPv6.
> >
> > I'm hoping that none of this (or the rest of the diff) is controversial,
> > but I wanted to make sure, so any review would be really appreciated.
> >
> > The patch is here:
> >
> >
> > http://people.freebsd.org/~markj/patches/network-providers/network-providers-1.diff
> >
> > It depends on r254468. To use it, just recompile the kernel (assuming
> > that KDTRACE_HOOKS is enabled) and copy
> > $SRC/cddl/lib/libdtrace/{ip,tcp,udp}.d (added by the patch) to
> > /usr/lib/dtrace. Then the example scripts I linked above should just
> > work.
> >
> 
> Great work Mark!
> 
> Here's some (from many) one-liners that this should make possible (more in
> the DTrace book):
> 
> Who is connecting to what:
> dtrace -n 'tcp:::accept-established { @[args[3]->tcps_raddr,
> args[3]->tcps_lport] = count(); }'
> 
> Who isn't connecting to what:
> dtrace -n 'tcp:::accept-refused { @[args[2]->ip_daddr, args[4]->tcp_sport]
> = count(); }'
> 
> What am I connecting to?
> dtrace -n 'tcp:::connect-established { @[args[3]->tcps_raddr ,
> args[3]->tcps_rport] = count(); }'
> 
> IP payload bytes for TCP send, size distribution by destination address:
> dtrace -n 'tcp:::send { @[args[2]->ip_daddr] =
> quantize(args[2]->ip_plength); }'
> 
> Many of these are very low overhead ways to understand network behavior -
> lower than sniffing every packet. Plus you can pull out kernel state that
> is never seen on the wire.
> 
> Brendan
> 
> -- 
> Brendan Gregg, Joyent                      http://dtrace.org/blogs/brendan