git: 434283fda99e - main - dwatch: add nine diagnostic modules; grow errno, io, proc, sched
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Tue, 28 Jul 2026 23:35:09 UTC
The branch main has been updated by dteske:
URL: https://cgit.FreeBSD.org/src/commit/?id=434283fda99e89af20c3fe95fde427624ae96d04
commit 434283fda99e89af20c3fe95fde427624ae96d04
Author: Devin Teske <dteske@FreeBSD.org>
AuthorDate: 2026-07-28 23:32:57 +0000
Commit: Devin Teske <dteske@FreeBSD.org>
CommitDate: 2026-07-28 23:32:57 +0000
dwatch: add nine diagnostic modules; grow errno, io, proc, sched
Grow the module collection to answer, each with a single command, the
first questions asked when diagnosing a sick system: why is my
application stalling, where is the kernel fighting over locks, what
file could it not find, why is this process getting EPERM, what killed
my process, will that fatal signal actually leave a core behind, what
was my process stuck on, where is my kernel memory going, who is
creating or entering jails, is the network slow because TCP is
resending, how long did my thread wait to run, and is the disk itself
slow. Every module keeps to the house style: invocation-name
overloading through hard links, predicate-only D with inline lookup
tables (no if-statements), and stable providers only (syscall, proc,
sched, io, dtmalloc, and the lockstat, vfs, priv, and mib SDT
providers), so the modules remain drop-in compatible with older
releases (the one documented exception is noted below). No kernel
changes: new and extended profiles under cddl/usr.sbin/dwatch/libexec
plus one libdtrace inline table (priv.d).
slow (slow-fsync, slow-open, slow-read, slow-syscall, slow-write, or
any slow-NAME by new link) records syscall entry timestamps in
thread-local storage and prints, at return, any call whose latency
meets a threshold (DWATCH_SLOW_MS, default 100), naming the syscall,
the elapsed time to the microsecond, and any errno returned. The bare
profile watches a curated set of filesystem-related calls expected to
be fast; slow-syscall watches everything; unrecognized invocation
names fall through to syscall::NAME:return with the matching entry
probe derived mechanically from the return probe list.
lock (lock-adaptive, lock-block, lock-lockmgr, lock-rw, lock-spin,
lock-sx, lock-thread) rides the dtrace_lockstat(4) block and spin
probes, printing the held-off thread (free from the standard event
tag), the holdoff duration, the lock class, the lo_name of the lock
through a single cast of arg0 to struct lock_object (the first member
of every kernel lock), and reader/writer intent on the probes that
report it. Holdoffs shorter than DWATCH_LOCK_MS (default 1; 0 shows
everything) are suppressed.
namei (namei-enoent, namei-entry, namei-failure) records the pathname
at vfs:namei:lookup:entry and reports it with the result at return.
Unlike the vop_lookup profile, which reconstructs paths from the name
cache one component at a time, this sees the whole path exactly as the
process requested it. namei-enoent hunts file-not-found storms -- the
single most common use of truss(1) -- without stopping the victim.
priv (priv-err, priv-ok) watches priv_check(9) verdicts, naming the
exact privilege denied -- something no syscall tracer can see, because
by the time EPERM surfaces the priv(9) value is gone. The number is
decoded by priv_string[], a new libdtrace inline table in the errno.d
and signal.d tradition, mechanically generated from sys/priv.h (247
entries) and installed to /usr/lib/dtrace where dtrace(1) auto-loads
it; on older releases it is a drop-in file like the module itself.
coredump (coredump-top) watches for delivery of signals whose default
action produces a core, per the SIGPROP_CORE entries of the sigproptbl
in kern_sig.c, and renders a verdict the same way and in the same
order the kernel will decide it: ignored or caught per the target's
struct sigacts, then the coredump() gauntlet of kern.coredump,
kern.sugid_coredump vs P_SUGID, procctl(2) PROC_TRACE_CTL, and
RLIMIT_CORE -- the sysctl knobs read live through kernel globals.
Where a coredump-worthy signal will produce no core, the verdict says
precisely which policy ate it. coredump-top maintains a cumulative
catalog of coredump-worthy signals by process and signal, refreshed
every 3 seconds in the style of systop; combine the event profile with
`-O cmd' to capture state as each event occurs.
hang (hang-top) pairs sched:::sleep with sched:::wakeup through a
tid-keyed timestamp array and prints, as each thread wakes, any sleep
that meets a threshold (DWATCH_HANG_MS, default 1000), naming the
sleeper in the details and the waker in the standard event tag. This
is the blocking the slow module structurally cannot see: a syscall
that never returns never reports its latency, while hang reports the
moment the wait ends, with the full duration. hang-top maintains a
cumulative catalog of long sleeps by process (count and maximum) in
the style of coredump-top.
jail (jail-attach, jail-get, jail-remove, jail-set) watches the jail
management plane -- jail(2), jail_set(2), jail_get(2), jail_attach(2),
and jail_remove(2) -- naming the operation, the jail id (taken from
the entry argument for attach/remove, from the return value for the
others), and any errno. Complements the dwatch `-j jail' filter,
which scopes any profile to processes inside one jail; this watches
who manipulates jails, from any jail or none.
dtmalloc (dtmalloc-top, or any dtmalloc-NAME by new link) rides the
dtmalloc provider (one malloc and one free probe per malloc(9) type).
The event profile prints allocations and frees meeting a size
threshold (DWATCH_MALLOC_MIN, default 65536) -- who is allocating huge
kernel buffers. dtmalloc-top maintains a running catalog of net bytes
and outstanding allocation balance by type, sorted by net bytes so
leak suspects rise: a type that climbs without bound while the system
is in steady state is the suspect. The catalog reflects activity
since the watch began, and is honest about caches holding what they
allocate.
mib (tcp-retransmit, or any mib-NAME by new link) rides the
per-counter mib SDT probes of the network stack. The tcp-retransmit
profile curates the counters that signal send-path congestion or loss
-- data packet retransmissions, unnecessary retransmissions,
retransmit timer expirations, and connections dropped by retransmit
exhaustion -- decoded through an inline description table, answering
"is this network slow because TCP is resending?" as events with
process context rather than netstat(1) deltas. NB: the mib probes
exist only in kernels built with options KDTRACE_MIB_SDT (default in
-CURRENT via std.debug); the module documents this and dtrace(1)
refuses the script elsewhere, making the dependency self-announcing.
Four existing modules gain personalities. proc grows
proc-signal-fatal, filtering signal-send to signals whose default
disposition terminates the receiver, most-notably including
kernel-generated SIGSEGV/SIGBUS/SIGILL/SIGFPE that no kill(2) watcher
will ever see. errno now reads its invocation name: errno-NAME shows
only syscalls returning that errno, where NAME is a symbolic name from
errno.d or a number; links are installed for errno-EACCES,
errno-ECAPMODE, errno-ENOENT, errno-ENOTCAPABLE, and errno-EPERM (the
latter pairs covering capsicum(4) capability-mode violations), and any
other errno needs only a new link. sched grows sched-latency,
recording a timestamp at sched:::enqueue keyed by tid and printing at
sched:::on-cpu any run-queue wait meeting a threshold (DWATCH_SCHED_MS,
default 10) -- the literal measurement of scheduler delay on a system
with idle CPU that still feels sluggish. io grows io-slow, pairing
io:::start with io:::done through a bio-keyed timestamp array and
printing any request that meets a threshold (DWATCH_IO_MS, default
100), naming the device, command, size, and elapsed time; watched
against zvols and a pool's leaf vdevs this brackets where in a ZFS
stack the time is going, without touching unstable providers.
Document all of the above plus the DWATCH_HANG_MS, DWATCH_IO_MS,
DWATCH_LOCK_MS, DWATCH_MALLOC_MIN, DWATCH_SCHED_MS, and DWATCH_SLOW_MS
knobs in dwatch(1).
All 46 new invocation names were exercised through `dwatch -d' with a
profile-path sandbox emulating the installed hard links: every one
sources cleanly and emits the intended D -- probe selection per alias,
entry/return and sleep/wakeup pairing through thread-local and global
associative arrays, threshold and mask predicates picking up their
knobs, aggregation clauses and printa column layout in the -top
profiles, multi-line predicate rendering, and `-t' correctly
displacing each module's default test were verified by inspection of
the generated scripts. Invocations untouched by this pass generate D
identical to their previous output. Modules pass sh -n, fit 80
columns, and dwatch.1 passes mandoc -Tlint with no new warnings. A
validation harness performing a `dwatch -e' compile per profile
against the live kernel globs every staged profile for runs wherever
the dtrace device is present.
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D58093
---
cddl/lib/libdtrace/Makefile | 6 +-
cddl/lib/libdtrace/mkpriv | 119 +++++++++++++++++++
cddl/usr.sbin/dwatch/dwatch.1 | 105 ++++++++++++++++-
cddl/usr.sbin/dwatch/libexec/Makefile | 42 +++++++
cddl/usr.sbin/dwatch/libexec/coredump | 210 ++++++++++++++++++++++++++++++++++
cddl/usr.sbin/dwatch/libexec/dtmalloc | 146 +++++++++++++++++++++++
cddl/usr.sbin/dwatch/libexec/errno | 15 ++-
cddl/usr.sbin/dwatch/libexec/hang | 174 ++++++++++++++++++++++++++++
cddl/usr.sbin/dwatch/libexec/io | 72 +++++++++++-
cddl/usr.sbin/dwatch/libexec/jail | 92 +++++++++++++++
cddl/usr.sbin/dwatch/libexec/lock | 136 ++++++++++++++++++++++
cddl/usr.sbin/dwatch/libexec/mib | 78 +++++++++++++
cddl/usr.sbin/dwatch/libexec/namei | 84 ++++++++++++++
cddl/usr.sbin/dwatch/libexec/priv | 59 ++++++++++
cddl/usr.sbin/dwatch/libexec/proc | 23 +++-
cddl/usr.sbin/dwatch/libexec/sched | 65 ++++++++++-
cddl/usr.sbin/dwatch/libexec/slow | 131 +++++++++++++++++++++
17 files changed, 1544 insertions(+), 13 deletions(-)
diff --git a/cddl/lib/libdtrace/Makefile b/cddl/lib/libdtrace/Makefile
index fbc6d5fbbec2..08836a91b164 100644
--- a/cddl/lib/libdtrace/Makefile
+++ b/cddl/lib/libdtrace/Makefile
@@ -55,6 +55,7 @@ DSRCS= cam.d \
errno.d \
io.d \
ip.d \
+ priv.d \
psinfo.d \
sctp.d \
siftr.d \
@@ -150,7 +151,7 @@ LDFLAGS+= -fsanitize=address -fsanitize=undefined
LIBADD= ctf elf proc pthread rtld_db xo
-CLEANFILES= dt_errtags.c dt_names.c
+CLEANFILES= dt_errtags.c dt_names.c priv.d
.include <bsd.lib.mk>
@@ -159,3 +160,6 @@ dt_errtags.c: ${OPENSOLARIS_USR_DISTDIR}/lib/libdtrace/common/dt_errtags.h
dt_names.c: ${OPENSOLARIS_SYS_DISTDIR}/uts/common/sys/dtrace.h
sh ${OPENSOLARIS_USR_DISTDIR}/lib/libdtrace/common/mknames.sh < ${.ALLSRC} > ${.TARGET}
+
+priv.d: mkpriv ${SRCTOP}/sys/sys/priv.h
+ sh ${.CURDIR}/mkpriv -o ${.TARGET} ${SRCTOP}/sys/sys/priv.h
diff --git a/cddl/lib/libdtrace/mkpriv b/cddl/lib/libdtrace/mkpriv
new file mode 100755
index 000000000000..d213161f8356
--- /dev/null
+++ b/cddl/lib/libdtrace/mkpriv
@@ -0,0 +1,119 @@
+#!/bin/sh
+# -*- tab-width: 4 -*- ;; Emacs
+# vi: set filetype=sh tabstop=8 shiftwidth=8 noexpandtab :: Vi/ViM
+############################################################ LICENSE
+#
+# Copyright (c) 2026 Devin Teske <dteske@FreeBSD.org>
+#
+# SPDX-License-Identifier: BSD-2-Clause
+#
+############################################################ IDENT(1)
+#
+# $Title: Generate priv.d from <sys/priv.h> for libdtrace(3) $
+# $Copyright: 2026 Devin Teske. All rights reserved. $
+#
+############################################################ GLOBALS
+
+VERSION='$Version: 1.0 $'
+
+pgm="${0##*/}" # Program basename
+
+#
+# Global exit status
+#
+SUCCESS=0
+FAILURE=1
+
+#
+# Command-line options
+#
+OUTPUT_FILE= # -o file
+
+#
+# Miscellaneous
+#
+PRIV_H=
+
+############################################################ FUNCTIONS
+
+usage()
+{
+ local fmt="$1"
+ local optfmt="\t%-5s %s\n"
+
+ exec >&2
+ if [ "$fmt" ]; then
+ shift 1 # fmt
+ printf "%s: $fmt\n" "$pgm" "$@"
+ fi
+ printf "Usage: %s [-hv] [-o file] priv.h\n" "$pgm"
+ printf "Options:\n"
+ printf "$optfmt" "-h" "Print this usage statment and exit."
+ printf "$optfmt" "-o file" "Write to output file instead of stdout."
+ printf "$optfmt" "-v" "Print version information and exit."
+ exit $FAILURE
+}
+
+############################################################ MAIN
+
+#
+# Process command-line options
+#
+while getopts ho:v flag; do
+ case "$flag" in
+ o) OUTPUT_FILE="$OPTARG" ;;
+ v) VERSION="${VERSION#*: }"
+ echo "${VERSION% $}"
+ exit $SUCCESS ;;
+ *) usage # NOTREACHED
+ esac
+done
+shift $(( $OPTIND - 1 ))
+
+#
+# Check command-line arguments
+#
+[ $# -gt 0 ] || usage "Too few arguments" # NOTREACHED
+[ $# -eq 1 ] || usage "Too many arguments" # NOTREACHED
+[ "$1" ] || usage "Missing priv.h argument" # NOTREACHED
+[ -e "$1" ] || usage "%s: No such file or directory" "$1" # NOTREACHED
+
+#
+# Generate priv.d contents from priv.h input
+#
+PRIV_H="$1"
+exec 9<<PRIV_D_EOF
+/*-
+ * Copyright (c) 2026 Devin Teske <dteske@FreeBSD.org>
+ *
+ * SPDX-License-Identifier: BSD-2-Clause
+ */
+
+/*
+ * Map priv(9) privilege values to their symbolic names.
+ * NB: Mechanically generated from <sys/priv.h> -- keep in-sync.
+ * NB: Generated by /usr/src/cddl/lib/libdtrace/$pgm
+ */
+
+#pragma D binding "1.13" priv_string
+inline string priv_string[int priv] =
+$( awk '
+ /^#define[[:space:]]+PRIV_[A-Z0-9_]+[[:space:]]+[0-9]+/ {
+ for (i = 0; ++i <= NF; ) if ($i ~ /define/) break
+ name = $++i
+ value = $++i
+ printf "\tpriv == %s ?\t\t\"%s\" :\n", value, name
+ }' "$PRIV_H"
+)
+ strjoin("PRIV_UNKNOWN#", lltostr(priv));
+PRIV_D_EOF
+
+#
+# Output contents
+#
+[ ! "$OUTPUT_FILE" ] || exec > "$OUTPUT_FILE"
+cat <&9
+
+################################################################################
+# END
+################################################################################
diff --git a/cddl/usr.sbin/dwatch/dwatch.1 b/cddl/usr.sbin/dwatch/dwatch.1
index 81efb2c2c437..8c201d970620 100644
--- a/cddl/usr.sbin/dwatch/dwatch.1
+++ b/cddl/usr.sbin/dwatch/dwatch.1
@@ -1,4 +1,4 @@
-.\" Copyright (c) 2014-2018 Devin Teske
+.\" Copyright (c) 2014-2026 Devin Teske
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
@@ -22,7 +22,7 @@
.\" ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
.\" POSSIBILITY OF SUCH DAMAGE.
.\"
-.Dd April 18, 2025
+.Dd July 4, 2026
.Dt DWATCH 1
.Os
.Sh NAME
@@ -382,17 +382,69 @@ Print mode and path from
.Xr chmod 2 ,
.Xr lchmod 2 ,
.Xr fchmodat 2
+.It coredump
+Print coredump-worthy signal deliveries with a verdict of whether a
+core will actually be written
+.It coredump-top
+Maintain a running catalog of coredump-worthy signals by process and
+signal
+.It dtmalloc
+Print kernel
+.Xr malloc 9
+and
+.Xr free 9
+activity by malloc type,
+meeting a size threshold
+.It dtmalloc-top
+Maintain a running catalog of net bytes and allocation balance by
+malloc type
+.Pq leak suspects rise
.It errno
Print non-zero errno results from system calls
+.It errno- Ns Ar NAME
+Print system calls returning the given errno,
+by symbolic name or number
+.Pq e.g., errno-ENOENT, errno-ENOTCAPABLE, errno-13
+.It hang
+Print threads that were asleep longer than a threshold,
+as they wake,
+naming the sleeper and the waker
+.It hang-top
+Maintain a running catalog of long sleeps by process
.It io
Print disk I/O details provided by
.Xr dtrace_io 4
+.It io-slow
+Print disk I/O requests exceeding a latency threshold
.It ip
Print IPv4 and IPv6 details provided by
.Xr dtrace_ip 4
+.It jail
+Print jail management activity from
+.Xr jail 2 ,
+.Xr jail_set 2 ,
+.Xr jail_get 2 ,
+.Xr jail_attach 2 ,
+.Xr jail_remove 2
.It kill
Print signal and pid from
.Xr kill 2
+.It lock
+Print kernel lock contention details provided by
+.Xr dtrace_lockstat 4
+.It lock-spin
+Print kernel lock spinning details provided by
+.Xr dtrace_lockstat 4
+.It mib
+Print network stack statistics counters as they increment
+.Pq requires a kernel built with options KDTRACE_MIB_SDT
+.It namei
+Print pathname resolution details provided by the
+.Fn vfs:namei:lookup
+probes
+.It namei-failure
+Print failed pathname resolutions
+.Pq also namei-enoent for ENOENT only
.It nanosleep
Print requested time from
.Xr nanosleep 2
@@ -400,12 +452,19 @@ Print requested time from
Print path from
.Xr open 2 ,
.Xr openat 2
+.It priv
+Print privilege requests denied by
+.Xr priv_check 9
+.Pq also priv-ok for those granted
.It proc
Print process execution details provided by
.Xr dtrace_proc 4
.It proc-signal
Print process signal details provided by
.Xr dtrace_proc 4
+.It proc-signal-fatal
+Print only signals whose default disposition is to terminate the
+receiving process
.It rw
Print buffer contents from
.Xr read 2 ,
@@ -413,12 +472,24 @@ Print buffer contents from
.It sched
Print CPU scheduling details provided by
.Xr dtrace_sched 4
+.It sched-latency
+Print threads that waited longer than a threshold between becoming
+runnable and running
+.It slow
+Print filesystem-related syscalls exceeding a latency threshold
+.It slow-syscall
+Print any syscall exceeding a latency threshold
.It tcp
Print TCP address/port details provided by
.Xr dtrace_tcp 4
.It tcp-io
Print TCP I/O details provided by
.Xr dtrace_tcp 4
+.It tcp-retransmit
+Print TCP retransmissions,
+retransmit timeouts,
+and connections dropped by retransmit exhaustion
+.Pq requires a kernel built with options KDTRACE_MIB_SDT
.It udp
Print UDP I/O details provided by
.Xr dtrace_udp 4
@@ -454,6 +525,26 @@ Print symlink paths being created by
These environment variables affect the execution of
.Nm :
.Bl -tag -width "DWATCH_PROFILES_PATH"
+.It Ev DWATCH_HANG_MS
+Minimum sleep duration,
+in milliseconds,
+displayed by the hang profiles
+.Pq default 1000; 0 to show everything .
+.It Ev DWATCH_IO_MS
+Minimum disk I/O request latency,
+in milliseconds,
+displayed by the io-slow profile
+.Pq default 100; 0 to show everything .
+.It Ev DWATCH_LOCK_MS
+Minimum lock holdoff duration,
+in milliseconds,
+displayed by the lock profiles
+.Pq default 1; 0 to show everything .
+.It Ev DWATCH_MALLOC_MIN
+Minimum allocation size,
+in bytes,
+displayed by the dtmalloc profiles
+.Pq default 65536; 0 to show everything .
.It Ev DWATCH_PROFILES_PATH
If
.Ev DWATCH_PROFILES_PATH
@@ -464,6 +555,16 @@ variable instead of the default
.Ql Li /usr/libexec/dwatch:/usr/local/libexec/dwatch .
If set to NULL,
profiles are not loaded.
+.It Ev DWATCH_SCHED_MS
+Minimum run-queue wait,
+in milliseconds,
+displayed by the sched-latency profile
+.Pq default 10; 0 to show everything .
+.It Ev DWATCH_SLOW_MS
+Minimum syscall latency,
+in milliseconds,
+displayed by the slow profiles
+.Pq default 100 .
.El
.Sh EXIT STATUS
.Ex -std
diff --git a/cddl/usr.sbin/dwatch/libexec/Makefile b/cddl/usr.sbin/dwatch/libexec/Makefile
index 72ea5be686fb..d32c54c34c83 100644
--- a/cddl/usr.sbin/dwatch/libexec/Makefile
+++ b/cddl/usr.sbin/dwatch/libexec/Makefile
@@ -1,16 +1,25 @@
PACKAGE= dwatch
FILESDIR= ${LIBEXECDIR}/dwatch
FILES= chmod \
+ coredump \
+ dtmalloc \
errno \
+ hang \
io \
ip \
+ jail \
kill \
+ lock \
+ mib \
+ namei \
nanosleep \
open \
+ priv \
proc \
rw \
sched \
sendrecv \
+ slow \
systop \
tcp \
udp \
@@ -23,11 +32,37 @@ FILES= chmod \
LINKMODE= ${SHAREMODE}
LINKS= ${LIBEXECDIR}/dwatch/chmod ${LIBEXECDIR}/dwatch/fchmodat
LINKS+= ${LIBEXECDIR}/dwatch/chmod ${LIBEXECDIR}/dwatch/lchmod
+LINKS+= ${LIBEXECDIR}/dwatch/coredump ${LIBEXECDIR}/dwatch/coredump-top
+LINKS+= ${LIBEXECDIR}/dwatch/dtmalloc ${LIBEXECDIR}/dwatch/dtmalloc-top
+LINKS+= ${LIBEXECDIR}/dwatch/errno ${LIBEXECDIR}/dwatch/errno-EACCES
+LINKS+= ${LIBEXECDIR}/dwatch/errno ${LIBEXECDIR}/dwatch/errno-ECAPMODE
+LINKS+= ${LIBEXECDIR}/dwatch/errno ${LIBEXECDIR}/dwatch/errno-ENOENT
+LINKS+= ${LIBEXECDIR}/dwatch/errno ${LIBEXECDIR}/dwatch/errno-ENOTCAPABLE
+LINKS+= ${LIBEXECDIR}/dwatch/errno ${LIBEXECDIR}/dwatch/errno-EPERM
+LINKS+= ${LIBEXECDIR}/dwatch/hang ${LIBEXECDIR}/dwatch/hang-top
LINKS+= ${LIBEXECDIR}/dwatch/io ${LIBEXECDIR}/dwatch/io-done
+LINKS+= ${LIBEXECDIR}/dwatch/io ${LIBEXECDIR}/dwatch/io-slow
LINKS+= ${LIBEXECDIR}/dwatch/io ${LIBEXECDIR}/dwatch/io-start
LINKS+= ${LIBEXECDIR}/dwatch/ip ${LIBEXECDIR}/dwatch/ip-receive
LINKS+= ${LIBEXECDIR}/dwatch/ip ${LIBEXECDIR}/dwatch/ip-send
+LINKS+= ${LIBEXECDIR}/dwatch/jail ${LIBEXECDIR}/dwatch/jail-attach
+LINKS+= ${LIBEXECDIR}/dwatch/jail ${LIBEXECDIR}/dwatch/jail-get
+LINKS+= ${LIBEXECDIR}/dwatch/jail ${LIBEXECDIR}/dwatch/jail-remove
+LINKS+= ${LIBEXECDIR}/dwatch/jail ${LIBEXECDIR}/dwatch/jail-set
+LINKS+= ${LIBEXECDIR}/dwatch/lock ${LIBEXECDIR}/dwatch/lock-adaptive
+LINKS+= ${LIBEXECDIR}/dwatch/lock ${LIBEXECDIR}/dwatch/lock-block
+LINKS+= ${LIBEXECDIR}/dwatch/lock ${LIBEXECDIR}/dwatch/lock-lockmgr
+LINKS+= ${LIBEXECDIR}/dwatch/lock ${LIBEXECDIR}/dwatch/lock-rw
+LINKS+= ${LIBEXECDIR}/dwatch/lock ${LIBEXECDIR}/dwatch/lock-spin
+LINKS+= ${LIBEXECDIR}/dwatch/lock ${LIBEXECDIR}/dwatch/lock-sx
+LINKS+= ${LIBEXECDIR}/dwatch/lock ${LIBEXECDIR}/dwatch/lock-thread
+LINKS+= ${LIBEXECDIR}/dwatch/mib ${LIBEXECDIR}/dwatch/tcp-retransmit
+LINKS+= ${LIBEXECDIR}/dwatch/namei ${LIBEXECDIR}/dwatch/namei-enoent
+LINKS+= ${LIBEXECDIR}/dwatch/namei ${LIBEXECDIR}/dwatch/namei-entry
+LINKS+= ${LIBEXECDIR}/dwatch/namei ${LIBEXECDIR}/dwatch/namei-failure
LINKS+= ${LIBEXECDIR}/dwatch/open ${LIBEXECDIR}/dwatch/openat
+LINKS+= ${LIBEXECDIR}/dwatch/priv ${LIBEXECDIR}/dwatch/priv-err
+LINKS+= ${LIBEXECDIR}/dwatch/priv ${LIBEXECDIR}/dwatch/priv-ok
LINKS+= ${LIBEXECDIR}/dwatch/proc ${LIBEXECDIR}/dwatch/proc-create
LINKS+= ${LIBEXECDIR}/dwatch/proc ${LIBEXECDIR}/dwatch/proc-exec
LINKS+= ${LIBEXECDIR}/dwatch/proc ${LIBEXECDIR}/dwatch/proc-exec-failure
@@ -36,6 +71,7 @@ LINKS+= ${LIBEXECDIR}/dwatch/proc ${LIBEXECDIR}/dwatch/proc-exit
LINKS+= ${LIBEXECDIR}/dwatch/proc ${LIBEXECDIR}/dwatch/proc-signal
LINKS+= ${LIBEXECDIR}/dwatch/proc ${LIBEXECDIR}/dwatch/proc-signal-clear
LINKS+= ${LIBEXECDIR}/dwatch/proc ${LIBEXECDIR}/dwatch/proc-signal-discard
+LINKS+= ${LIBEXECDIR}/dwatch/proc ${LIBEXECDIR}/dwatch/proc-signal-fatal
LINKS+= ${LIBEXECDIR}/dwatch/proc ${LIBEXECDIR}/dwatch/proc-signal-send
LINKS+= ${LIBEXECDIR}/dwatch/proc ${LIBEXECDIR}/dwatch/proc-status
LINKS+= ${LIBEXECDIR}/dwatch/rw ${LIBEXECDIR}/dwatch/read
@@ -45,6 +81,7 @@ LINKS+= ${LIBEXECDIR}/dwatch/sched ${LIBEXECDIR}/dwatch/sched-cpu
LINKS+= ${LIBEXECDIR}/dwatch/sched ${LIBEXECDIR}/dwatch/sched-dequeue
LINKS+= ${LIBEXECDIR}/dwatch/sched ${LIBEXECDIR}/dwatch/sched-enqueue
LINKS+= ${LIBEXECDIR}/dwatch/sched ${LIBEXECDIR}/dwatch/sched-exec
+LINKS+= ${LIBEXECDIR}/dwatch/sched ${LIBEXECDIR}/dwatch/sched-latency
LINKS+= ${LIBEXECDIR}/dwatch/sched ${LIBEXECDIR}/dwatch/sched-lend-pri
LINKS+= ${LIBEXECDIR}/dwatch/sched ${LIBEXECDIR}/dwatch/sched-load-change
LINKS+= ${LIBEXECDIR}/dwatch/sched ${LIBEXECDIR}/dwatch/sched-off-cpu
@@ -63,6 +100,11 @@ LINKS+= ${LIBEXECDIR}/dwatch/sendrecv ${LIBEXECDIR}/dwatch/recvmsg
LINKS+= ${LIBEXECDIR}/dwatch/sendrecv ${LIBEXECDIR}/dwatch/send
LINKS+= ${LIBEXECDIR}/dwatch/sendrecv ${LIBEXECDIR}/dwatch/sendmsg
LINKS+= ${LIBEXECDIR}/dwatch/sendrecv ${LIBEXECDIR}/dwatch/sendto
+LINKS+= ${LIBEXECDIR}/dwatch/slow ${LIBEXECDIR}/dwatch/slow-fsync
+LINKS+= ${LIBEXECDIR}/dwatch/slow ${LIBEXECDIR}/dwatch/slow-open
+LINKS+= ${LIBEXECDIR}/dwatch/slow ${LIBEXECDIR}/dwatch/slow-read
+LINKS+= ${LIBEXECDIR}/dwatch/slow ${LIBEXECDIR}/dwatch/slow-syscall
+LINKS+= ${LIBEXECDIR}/dwatch/slow ${LIBEXECDIR}/dwatch/slow-write
LINKS+= ${LIBEXECDIR}/dwatch/tcp ${LIBEXECDIR}/dwatch/tcp-accept
LINKS+= ${LIBEXECDIR}/dwatch/tcp ${LIBEXECDIR}/dwatch/tcp-accept-established
LINKS+= ${LIBEXECDIR}/dwatch/tcp ${LIBEXECDIR}/dwatch/tcp-accept-refused
diff --git a/cddl/usr.sbin/dwatch/libexec/coredump b/cddl/usr.sbin/dwatch/libexec/coredump
new file mode 100644
index 000000000000..0b68564691de
--- /dev/null
+++ b/cddl/usr.sbin/dwatch/libexec/coredump
@@ -0,0 +1,210 @@
+# -*- tab-width: 4 -*- ;; Emacs
+# vi: set filetype=sh tabstop=8 shiftwidth=8 noexpandtab :: Vi/ViM
+############################################################ IDENT(1)
+#
+# $Title: dwatch(8) module for coredump-worthy signal delivery $
+# $Copyright: 2026 Devin Teske. All rights reserved. $
+#
+############################################################ DESCRIPTION
+#
+# Print signals whose default action produces a coredump, as they are
+# sent, together with a verdict of whether a core will actually be
+# written, rendered the same way and in the same order the kernel will
+# decide it: the target may be ignoring or catching the signal, dumps
+# may be disabled (kern.coredump), the target may be flagged sugid
+# (kern.sugid_coredump) or may have disabled tracing via procctl(2)
+# PROC_TRACE_CTL, or RLIMIT_CORE may be 0.
+# Coredump-worthy is defined by the SIGPROP_CORE entries of the
+# sigproptbl in kern_sig.c: SIGQUIT, SIGILL, SIGTRAP, SIGABRT, SIGEMT,
+# SIGFPE, SIGBUS, SIGSEGV, and SIGSYS.
+# The coredump-top profile maintains a running catalog, updated
+# every 3 seconds, of coredump-worthy signals by process and signal.
+# Combine with `-O cmd' to capture state as each event occurs.
+#
+############################################################ PRAGMAS
+
+case "$PROFILE" in
+coredump-top)
+ DTRACE_PRAGMA="
+ option quiet
+ option aggsortrev
+ " # END-QUOTE
+ ;;
+esac
+
+############################################################ PROBE
+
+case "$PROFILE" in
+coredump-top)
+ : ${PROBE:=profile:::tick-3s} ;;
+*)
+ : ${PROBE:=proc:::signal-send}
+esac
+
+############################################################ GLOBALS
+
+exec 9<<EOF
+/*
+ * Signals whose default action includes a coredump, per the
+ * SIGPROP_CORE entries of the sigproptbl in kern_sig.c of FreeBSD
+ */
+inline int coredump_sig[int sig] =
+ sig == SIGQUIT ? 1 :
+ sig == SIGILL ? 1 :
+ sig == SIGTRAP ? 1 :
+ sig == SIGABRT ? 1 : /* SIGIOT */
+ sig == SIGEMT ? 1 :
+ sig == SIGFPE ? 1 :
+ sig == SIGBUS ? 1 :
+ sig == SIGSEGV ? 1 :
+ sig == SIGSYS ? 1 :
+ 0;
+EOF
+GLOBALS=$( cat <&9 )
+
+############################################################ EVENT ACTION
+
+[ "$CUSTOM_TEST" ] || case "$PROFILE" in
+coredump-top) ;;
+*) EVENT_TEST="coredump_sig[this->sig]"
+esac
+
+############################################################ ACTIONS
+
+if [ "$PROFILE" = "coredump-top" ]; then
+exec 9<<EOF
+$GLOBALS
+
+BEGIN { printf("Cataloging coredump-worthy signals ...") } /* probe ID $ID */
+
+proc:::signal-send /coredump_sig[(int)arg2]/ /* probe ID $(( $ID + 1 )) */
+{
+ @cores[stringof(((struct proc *)args[1])->p_comm),
+ signal_string[(int)arg2]] = count();
+}
+EOF
+ACTIONS=$( cat <&9 )
+ID=$(( $ID + 2 ))
+else
+exec 9<<EOF
+$GLOBALS
+
+this int sig;
+this pid_t pid;
+this string verdict;
+this struct proc * target;
+this u_int core_catch;
+this u_int core_ign;
+this u_int core_notrace;
+this u_int core_sugid;
+this uint64_t core_limit;
+
+$PROBE /* probe ID $ID */
+{${TRACE:+
+ printf("<$ID>");
+}
+ this->target = (struct proc *)args[1];
+ this->pid = (pid_t)this->target->p_pid;
+ this->sig = (int)arg2;
+}
+
+$PROBE /* probe ID $(( $ID + 1 )) */
+{${TRACE:+
+ printf("<$(( $ID + 1 ))>");
+}
+ /*
+ * Render a verdict the way the kernel will: disposition first
+ * (struct sigacts), then the coredump() gauntlet in its order:
+ * kern.coredump, kern.sugid_coredump vs P_SUGID, procctl(2)
+ * PROC_TRACE_CTL (P2_NOTRACE), and finally RLIMIT_CORE
+ */
+ this->core_ign =
+ this->target->p_sigacts->ps_sigignore.__bits[
+ (this->sig - 1) >> 5] &
+ (1 << ((this->sig - 1) & 31));
+ this->core_catch =
+ this->target->p_sigacts->ps_sigcatch.__bits[
+ (this->sig - 1) >> 5] &
+ (1 << ((this->sig - 1) & 31));
+ this->core_sugid =
+ this->target->p_flag & 0x00000100; /* P_SUGID */
+ this->core_notrace =
+ this->target->p_flag2 & 0x00000002; /* P2_NOTRACE */
+ this->core_limit = this->target->p_limit->
+ pl_rlimit[4].rlim_cur; /* RLIMIT_CORE */
+
+ this->verdict =
+ this->core_ign ? "ignored" :
+ this->core_catch ? "caught" :
+ \`do_coredump == 0 ? "denied by kern.coredump" :
+ this->core_sugid && \`sugid_coredump == 0 ?
+ "denied by kern.sugid_coredump" :
+ this->core_notrace ? "denied by procctl trace ctl" :
+ this->core_limit == 0 ? "denied by RLIMIT_CORE" :
+ "will dump core";
+
+ $( pproc -P _core "(struct proc *)args[1]" )
+}
+EOF
+ACTIONS=$( cat <&9 )
+ID=$(( $ID + 2 ))
+fi
+
+############################################################ EVENT TAG
+
+# For the running catalog, override the default `UID.GID CMD[PID]: ' tag
+# with ANSI cursor-homing and screen-clearing codes plus column headers.
+
+if [ "$PROFILE" = "coredump-top" ]; then
+size=$( stty size 2> /dev/null )
+rows="${size%% *}"
+cols="${size#* }"
+
+exec 9<<EOF
+ printf("\033[H"); /* Position the cursor at top-left */
+ printf("\033[J"); /* Clear display from cursor to end */
+
+ /* Header line containing probe (left) and date (right) */
+ printf("%-*s%s%Y%s\n",
+ $(( ${cols:-80} - 20 )), "$PROBE",
+ console ? "\033[32m" : "",
+ walltimestamp,
+ console ? "\033[39m" : "");
+
+ /* Column headers */
+ printf("%s%8s %-20s %s%s\n",
+ console ? "\033[1m" : "",
+ "COUNT",
+ "EXECNAME",
+ "SIGNAL",
+ console ? "\033[22m" : "");
+EOF
+EVENT_TAG=$( cat <&9 )
+fi
+
+############################################################ EVENT DETAILS
+
+if [ "$PROFILE" = "coredump-top" ]; then
+exec 9<<EOF
+ /* NB: Cumulative; not truncated between updates */
+ printa("%@8u %-20s %s\n", @cores);
+EOF
+EVENT_DETAILS=$( cat <&9 )
+elif [ ! "$CUSTOM_DETAILS" ]; then
+exec 9<<EOF
+ /*
+ * Print coredump-worthy signal details
+ */
+ printf("%s[%d] pid %d (%s) -- %s",
+ signal_string[this->sig],
+ this->sig,
+ this->pid,
+ this->verdict,
+ this->args_core);
+EOF
+EVENT_DETAILS=$( cat <&9 )
+fi
+
+################################################################################
+# END
+################################################################################
diff --git a/cddl/usr.sbin/dwatch/libexec/dtmalloc b/cddl/usr.sbin/dwatch/libexec/dtmalloc
new file mode 100644
index 000000000000..ad1a044020fa
--- /dev/null
+++ b/cddl/usr.sbin/dwatch/libexec/dtmalloc
@@ -0,0 +1,146 @@
+# -*- tab-width: 4 -*- ;; Emacs
+# vi: set filetype=sh tabstop=8 shiftwidth=8 noexpandtab :: Vi/ViM
+############################################################ IDENT(1)
+#
+# $Title: dwatch(8) module for malloc(9) type activity $
+# $Copyright: 2026 Devin Teske. All rights reserved. $
+#
+############################################################ DESCRIPTION
+#
+# Print kernel malloc(9) and free(9) activity by malloc type, via the
+# dtmalloc provider (one malloc and one free probe per type; see
+# vmstat -m for the types). The default profile prints allocations and
+# frees meeting a size threshold (default 65536 bytes; tunable via
+# DWATCH_MALLOC_MIN in the environment, 0 to show everything),
+# answering "who is allocating huge kernel buffers?" Use
+# dtmalloc-NAME to watch a single type (e.g., dtmalloc-devbuf).
+# The dtmalloc-top profile maintains a running catalog, updated every
+# 3 seconds, of net bytes and outstanding allocation balance by type:
+# a type whose net bytes climb without bound while the system is in
+# steady state is a leak suspect. NB: the catalog reflects activity
+# since the watch began, not preexisting allocations, and caches
+# legitimately hold what they allocate.
+#
+############################################################ PRAGMAS
+
+case "$PROFILE" in
+dtmalloc-top)
+ DTRACE_PRAGMA="
+ option quiet
+ option aggsortrev
+ " # END-QUOTE
+ ;;
+esac
+
+############################################################ PROBE
+
+case "$PROFILE" in
+dtmalloc)
+ : ${PROBE:=dtmalloc:::malloc, dtmalloc:::free} ;;
+dtmalloc-top)
+ : ${PROBE:=profile:::tick-3s} ;;
+*)
+ : ${PROBE:=$( echo \
+ dtmalloc::${PROFILE#dtmalloc-}:malloc, \
+ dtmalloc::${PROFILE#dtmalloc-}:free )} ;;
+esac
+
+############################################################ EVENT ACTION
+
+: ${DWATCH_MALLOC_MIN:=65536}
+
+case "$DWATCH_MALLOC_MIN" in
+""|*[!0-9]*) die "DWATCH_MALLOC_MIN must be a number" ;; # NOTREACHED
+esac
+
+[ "$CUSTOM_TEST" ] || case "$PROFILE" in
+dtmalloc-top) ;;
+*) EVENT_TEST="(uint64_t)arg3 >= $DWATCH_MALLOC_MIN"
+esac
+
+############################################################ ACTIONS
+
+if [ "$PROFILE" = "dtmalloc-top" ]; then
+exec 9<<EOF
+this int64_t dtmalloc_delta;
+
+BEGIN { printf("Cataloging malloc(9) activity ...") } /* probe ID $ID */
+
+dtmalloc:::malloc /* probe ID $(( $ID + 1 )) */
+{
+ this->dtmalloc_delta = (int64_t)arg3;
+}
+
+dtmalloc:::free /* probe ID $(( $ID + 2 )) */
+{
+ this->dtmalloc_delta = -(int64_t)arg3;
+}
+
+dtmalloc:::malloc, dtmalloc:::free /* probe ID $(( $ID + 3 )) */
+{
+ @dtmalloc_bytes[probefunc] = sum(this->dtmalloc_delta);
+ @dtmalloc_allocs[probefunc] =
+ sum(this->dtmalloc_delta > 0 ? 1 : -1);
+}
+EOF
+ACTIONS=$( cat <&9 )
+ID=$(( $ID + 4 ))
+fi
+
+############################################################ EVENT TAG
+
+# For the running catalog, override the default `UID.GID CMD[PID]: ' tag
+# with ANSI cursor-homing and screen-clearing codes plus column headers.
+
+if [ "$PROFILE" = "dtmalloc-top" ]; then
+size=$( stty size 2> /dev/null )
+rows="${size%% *}"
+cols="${size#* }"
+
+exec 9<<EOF
+ printf("\033[H"); /* Position the cursor at top-left */
+ printf("\033[J"); /* Clear display from cursor to end */
+
+ /* Header line containing probe (left) and date (right) */
+ printf("%-*s%s%Y%s\n",
+ $(( ${cols:-80} - 20 )), "$PROBE",
+ console ? "\033[32m" : "",
+ walltimestamp,
+ console ? "\033[39m" : "");
+
+ /* Column headers */
+ printf("%s%14s %10s %s%s\n",
+ console ? "\033[1m" : "",
+ "NET(B)",
+ "BALANCE",
+ "TYPE",
+ console ? "\033[22m" : "");
+EOF
+EVENT_TAG=$( cat <&9 )
+fi
+
+############################################################ EVENT DETAILS
+
+if [ "$PROFILE" = "dtmalloc-top" ]; then
+exec 9<<EOF
+ /* NB: Cumulative; not truncated between updates */
+ printa("%@14d %@10d %s\n", @dtmalloc_bytes, @dtmalloc_allocs);
+EOF
+EVENT_DETAILS=$( cat <&9 )
+elif [ ! "$CUSTOM_DETAILS" ]; then
+exec 9<<EOF
+ /*
+ * Print malloc(9) activity details
+ */
+ printf("%s(9) type %s %d byte%s",
+ probename,
+ probefunc,
+ (long)arg3,
+ (long)arg3 == 1 ? "" : "s");
+EOF
+EVENT_DETAILS=$( cat <&9 )
+fi
+
+################################################################################
+# END
+################################################################################
diff --git a/cddl/usr.sbin/dwatch/libexec/errno b/cddl/usr.sbin/dwatch/libexec/errno
index d899906a1313..e87cbde7ce50 100644
--- a/cddl/usr.sbin/dwatch/libexec/errno
+++ b/cddl/usr.sbin/dwatch/libexec/errno
@@ -3,7 +3,7 @@
############################################################ IDENT(1)
#
# $Title: dwatch(8) module for syscall errno logging $
-# $Copyright: 2014-2018 Devin Teske. All rights reserved. $
+# $Copyright: 2014-2026 Devin Teske. All rights reserved. $
#
############################################################ DESCRIPTION
#
@@ -11,13 +11,24 @@
*** 1079 LINES SKIPPED ***