[Bug 297335] dtrace(1): FBT probe on an NMI-reachable function re-enters dtrace_probe() and panics an INVARIANTS kernel

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=297335

            Bug ID: 297335
           Summary: dtrace(1): FBT probe on an NMI-reachable function
                    re-enters dtrace_probe() and panics an INVARIANTS
                    kernel
           Product: Base System
           Version: CURRENT
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Some People
          Priority: ---
         Component: kern
          Assignee: bugs@FreeBSD.org
          Reporter: netchild@FreeBSD.org

Description:
Instrumenting a function that is reachable from NMI context with FBT, while any
other probe is also enabled, panics an INVARIANTS kernel:

panic: assertion failed: curthread->t_dtrace_inprobe == 0 ||
    id == dtrace_probeid_error,
    file: sys/cddl/contrib/opensolaris/uts/common/dtrace/dtrace.c, line: 7323

hwpmc(4)'s sampling interrupt is the ordinary way to reach NMI context on a
FreeBSD machine, so the reproducer uses it, but nothing here is specific to
hwpmc — any NMI-reachable code has the same exposure.

Reproducer:
Base system only, no custom binaries, all as root (DTrace). On a GENERIC kernel
with INVARIANTS and device hwpmc:
-----
kldload hwpmc dtraceall

dtrace -q -n '
        fbt::pmc_add_sample:entry { @nmi = count(); }
        syscall:::entry           { @sys = count(); }
' &

pmcstat -P instructions -O /var/tmp/out.pmclog -t $$
-----
The machine panics within seconds.

Both clauses are needed, and that was checked rather than assumed. With only
the fbt::pmc_add_sample:entry clause the same load runs for 25 s without a
panic. An NMI cannot interrupt itself — further NMIs are blocked until IRET —
so the NMI-context probe alone has no window to nest in; it needs a probe
firing in ordinary context to nest into.

Backtrace:
The backtrace is the whole mechanism in one artifact — an outer probe in a
pipe2 syscall, an NMI on top of it, and FBT re-entering dtrace_probe() from the
NMI:

panic: assertion failed: curthread->t_dtrace_inprobe == 0 ||
    id == dtrace_probeid_error, .../dtrace/dtrace.c, line: 7323
db_trace_self_wrapper()
vpanic()
dtrace_panic()
dtrace_probe()            <-- nested probe, entered from NMI context
fbt_invop()
dtrace_invop()
dtrace_invop_start()
nmi_handle_intr()
nmi_calltrap()
--- trap 0x13, rip = ..., rsp = ..., rbp = ... ---
dtrace_probe()            <-- outer probe, ordinary context
systrace_probe()
amd64_syscall()
fast_syscall_common()
--- syscall (542, FreeBSD ELF64, pipe2) ---

Analysis:
dtrace_probe_enter() (dtrace.c:7307) guards against running a probe inside a
probe by calling dtrace_interrupt_disable() and then asserting that the
per-thread t_dtrace_inprobe flag is clear. An NMI is not maskable, so disabling
interrupts does not close that window. A PMC sampling interrupt arriving while
a thread is inside a probe runs pmc_process_interrupt() → pmc_add_sample(), and
if FBT has instrumented either of those, dtrace_probe() re-enters.

fbt_excluded() (sys/cddl/dev/fbt/fbt.c:105) already maintains exactly the right
kind of list for this — it refuses to instrument code reachable from probe
context: anything dtrace_* (except dtrace_safe_*), db_/kdb_, the lock owner
methods, the KMSAN runtime, the stack unwinders on some platforms, and fbt_*
when DTrace is built into the kernel. It has no case for code reachable from
NMI context: the others are reachable by a call path from a probe, whereas this
one arrives asynchronously and cannot be prevented by disabling interrupts.

The assertion appears to be doing exactly the job it was added for
(859313642848: "helps catch cases where an instrumented function is called
while in probe context").

Without INVARIANTS:
No panic: the assertion is ASSERT(), which dtrace_impl.h:1324 compiles only
#ifdef DEBUG, and sys/cddl/compat/opensolaris/sys/debug_compat.h defines DEBUG
exactly when INVARIANTS is set. Verified on a GENERIC-NODEBUG kernel + device
hwpmc: the same probes and the same load run to completion (240k interrupts
processed) and the machine stays up.

Whether the re-entry still happens there, unobserved, was not determined — the
only detector is the assertion that is compiled out. If it does, it is worth
noting that dtrace_probe_exit() sets t_dtrace_inprobe = 0 unconditionally
rather than saving and restoring it, so a nested probe would leave the outer
probe running with the flag clear; and the comment at dtrace.c:7315 says
re-entry means "the ordering guarantee of the records will be violated,
resulting in unexpected output".

Impact:
Low. DTrace requires root — /dev/dtrace/dtrace is crw------- root:wheel and an
unprivileged user gets "DTrace requires additional privileges" for every
operation, including dtrace -l — so this needs an administrator to be tracing
at the time. The workload half needs no privilege (a process-mode sampling PMC
on one's own process), but that alone is harmless: the same load with no DTrace
running survives, having processed 247k interrupts. So: a local denial of
service on a debug kernel, requiring root to be tracing. Reported as a
correctness problem, not a DoS one.

-- 
You are receiving this mail because:
You are the assignee for the bug.