git: aa3b7a2fbc46 - main - /etc/rc add trace debug and verify

From: Simon J. Gerraty <sjg_at_FreeBSD.org>
Date: Fri, 09 Feb 2024 17:17:28 UTC
The branch main has been updated by sjg:

URL: https://cgit.FreeBSD.org/src/commit/?id=aa3b7a2fbc4687c0a09b6166aa2c2d117989d8fa

commit aa3b7a2fbc4687c0a09b6166aa2c2d117989d8fa
Author:     Simon J. Gerraty <sjg@FreeBSD.org>
AuthorDate: 2024-02-09 17:15:58 +0000
Commit:     Simon J. Gerraty <sjg@FreeBSD.org>
CommitDate: 2024-02-09 17:15:58 +0000

    /etc/rc add trace debug and verify
    
    Debugging boot issues can be helped by
    logging each rc.d script as it is run
    and being able to selectively enable/disable set -x
    debug.sh provides an elaborate framework for debugging shell scripts.
    
    For secure systems, we want to be paranoid about what we read
    during boot.
    
    dot()   simply reads (.) arg file if it exists
    vdot()  if mac_veriexec is active, ignore unverified files
            otherwise behaves much the same as dot()
    safe_dot()  in safe_eval.sh allows reading an untrusted file;
            limiting the input to simple variable assignments.
    
    In load_rc_config allow caller to provide an option to indicate how to
    handle its arg:
            -v use vdot()
            -s use sdot() which will try to use vdot() and fallback to safe_dot()
            The default is to read using dot()
    
    rc_run_scripts()
            encapsulate the running of rc.d scripts
            so that we can easily call it more than twice.
    
    We vdot local.rc.subr to pick up extensions (like
    run_rc_scripts_final) and overrides.
    
    We also allow rc.subr.local or rc.conf to set rc_config_xtra
    eg (rc_config_xtra=XXX for historic compatibility)
    
    rc use set -o verify around the reading in of rc.subr
    This has no effect if mac_veriexec is not active, but if it is; ensures
    rc.subr has not been tampered with.
    
    Reviewed by:    imp
    Sponsored by:   Juniper Networks, Inc.
    Differential Revision:  https://reviews.freebsd.org/D43671
---
 libexec/rc/Makefile       |   6 +
 libexec/rc/debug.sh       | 278 ++++++++++++++++++++++++++++++++++++++++++++++
 libexec/rc/rc             |  28 ++---
 libexec/rc/rc.subr        | 233 +++++++++++++++++++++++++++++++++++++-
 libexec/rc/safe_eval.sh   |  66 +++++++++++
 share/man/man8/Makefile   |   1 +
 share/man/man8/debug.sh.8 | 182 ++++++++++++++++++++++++++++++
 share/man/man8/rc.8       |  18 ++-
 share/man/man8/rc.subr.8  | 220 +++++++++++++++++++++++++++++++++++-
 9 files changed, 1001 insertions(+), 31 deletions(-)

diff --git a/libexec/rc/Makefile b/libexec/rc/Makefile
index 8e42c12e1163..48115d873fe3 100644
--- a/libexec/rc/Makefile
+++ b/libexec/rc/Makefile
@@ -18,6 +18,12 @@ CONFETCDEFAULTSDIR=	/etc/defaults
 CONFETCDEFAULTS=	rc.conf
 CONFETCDEFAULTSPACKAGE=	rc
 
+FILESGROUPS=	LIBEXEC_SCRIPTS
+LIBEXEC_SCRIPTS=	debug.sh safe_eval.sh
+LIBEXEC_SCRIPTSDIR=	/libexec
+LIBEXEC_SCRIPTSMODE=	755
+LIBEXEC_SCRIPTSPACKAGE=	rc
+
 SUBDIR+=	rc.d
 
 HAS_TESTS=
diff --git a/libexec/rc/debug.sh b/libexec/rc/debug.sh
new file mode 100755
index 000000000000..7bbb500e2d22
--- /dev/null
+++ b/libexec/rc/debug.sh
@@ -0,0 +1,278 @@
+:
+# SPDX-License-Identifier: BSD-2-Clause
+
+# NAME:
+#	debug.sh - selectively debug scripts
+#
+# SYNOPSIS:
+#	$_DEBUG_SH . debug.sh
+#	DebugOn [-eo] "tag" ...
+#	DebugOff [-eo] [rc="rc"] "tag" ...
+#	Debugging
+#	DebugEcho ...
+#	DebugLog ...
+#	DebugShell "tag" ...
+#	DebugTrace ...
+#	Debug "tag" ...
+#
+#	$DEBUG_SKIP echo skipped when Debug "tag" is true.
+#	$DEBUG_DO echo only done when Debug "tag" is true.
+#
+# DESCRIPTION:
+#	debug.sh provides the following functions to facilitate
+#	flexible run-time tracing of complicated shell scripts.
+#
+#	DebugOn turns tracing on if any "tag" is found in "DEBUG_SH".
+#	It turns tracing off if "!tag" is found in "DEBUG_SH".
+#	It also sets "DEBUG_ON" to the "tag" that caused tracing to be
+#	enabled, or "DEBUG_OFF" if we matched "!tag".
+#	If '-e' option given returns 1 if no "tag" matched.
+#	If the '-o' flag is given, tracing is turned off unless there
+#	was a matched "tag", useful for functions too noisy to tace.
+#
+#	DebugOff turns tracing on if any "tag" matches "DEBUG_OFF" or
+#	off if any "tag" matches "DEBUG_ON". This allows nested
+#	functions to not interfere with each other.
+#
+#	DebugOff accepts but ignores the '-e' and '-o' options.
+#	The optional "rc" value will be returned rather than the
+#	default of 0. Thus if DebugOff is the last operation in a
+#	function, "rc" will be the return code of that function.
+#
+#	DebugEcho is just shorthand for:
+#.nf
+#	$DEBUG_DO echo "$@"
+#.fi
+#
+#	Debugging returns true if tracing is enabled.
+#	It is useful for bounding complex debug actions, rather than
+#	using lots of "DEBUG_DO" lines.
+#
+#	DebugShell runs an interactive shell if any "tag" is found in
+#	"DEBUG_INTERACTIVE", and there is a tty available.
+#	The shell used is defined by "DEBUG_SHELL" or "SHELL" and
+#	defaults to '/bin/sh'.
+#
+#	Debug calls DebugOn and if that does not turn tracing on, it
+#	calls DebugOff to turn it off.
+#
+#	The variables "DEBUG_SKIP" and "DEBUG_DO" are set so as to
+#	enable/disable code that should be skipped/run when debugging
+#	is turned on. "DEBUGGING" is the same as "DEBUG_SKIP" for
+#	backwards compatability.
+#
+#	The use of $_DEBUG_SH is to prevent multiple inclusion, though
+#	it does no harm in this case.
+#
+# BUGS:
+#	Does not work with some versions of ksh.
+#	If a function turns tracing on, ksh turns it off when the
+#	function returns - useless.
+#	PD ksh works ok ;-)
+#
+# AUTHOR:
+#	Simon J. Gerraty <sjg@crufty.net>
+
+# RCSid:
+#	$Id: debug.sh,v 1.35 2024/02/03 19:04:47 sjg Exp $
+#
+#	@(#) Copyright (c) 1994-2024 Simon J. Gerraty
+#
+#	This file is provided in the hope that it will
+#	be of use.  There is absolutely NO WARRANTY.
+#	Permission to copy, redistribute or otherwise
+#	use this file is hereby granted provided that
+#	the above copyright notice and this notice are
+#	left intact.
+#
+#	Please send copies of changes and bug-fixes to:
+#	sjg@crufty.net
+#
+
+_DEBUG_SH=:
+
+Myname=${Myname:-`basename $0 .sh`}
+
+DEBUGGING=
+DEBUG_DO=:
+DEBUG_SKIP=
+export DEBUGGING DEBUG_DO DEBUG_SKIP
+
+_debugOn() {
+	DEBUG_OFF=
+	DEBUG_DO=
+	DEBUG_SKIP=:
+	DEBUG_X=-x
+	set -x
+	DEBUG_ON=$1
+}
+
+_debugOff() {
+	DEBUG_OFF=$1
+	set +x
+	DEBUG_ON=$2
+	DEBUG_DO=:
+	DEBUG_SKIP=
+	DEBUG_X=
+}
+
+DebugEcho() {
+	$DEBUG_DO echo "$@"
+}
+
+Debugging() {
+	test "$DEBUG_SKIP"
+}
+
+DebugLog() {
+	$DEBUG_SKIP return 0
+	echo `date '+@ %s [%Y-%m-%d %H:%M:%S %Z]'` "$@"
+}
+
+# something hard to miss when wading through huge -x output
+DebugTrace() {
+	$DEBUG_SKIP return 0
+	set +x
+	echo "@ ==================== [ $DEBUG_ON ] ===================="
+	DebugLog "$@"
+	echo "@ ==================== [ $DEBUG_ON ] ===================="
+	set -x
+}
+
+# Turn on debugging if appropriate
+DebugOn() {
+	_rc=0			# avoid problems with set -e
+	_off=:
+	while :
+	do
+		case "$1" in
+		-e) _rc=1; shift;; # caller ok with return 1
+		-o) _off=; shift;; # off unless we have a match
+		*) break;;
+		esac
+	done
+	case ",${DEBUG_SH:-$DEBUG}," in
+	,,)	return $_rc;;
+	*,[Dd]ebug,*) ;;
+	*) $DEBUG_DO set +x;;		# reduce the noise
+	esac
+	_match=
+	# if debugging is off because of a !e
+	# don't add 'all' to the On list.
+	case "$_off$DEBUG_OFF" in
+	:)	_e=all;;
+	*)	_e=;;
+	esac
+	for _e in ${*:-$Myname} $_e
+	do
+		: $_e in ,${DEBUG_SH:-$DEBUG},
+		case ",${DEBUG_SH:-$DEBUG}," in
+		*,!$_e,*|*,!$Myname:$_e,*)
+			# only turn it off if it was on
+			_rc=0
+			$DEBUG_DO _debugOff $_e $DEBUG_ON
+			break
+			;;
+		*,$_e,*|*,$Myname:$_e,*)
+			# only turn it on if it was off
+			_rc=0
+			_match=$_e
+			$DEBUG_SKIP _debugOn $_e
+			break
+			;;
+		esac
+	done
+	if test -z "$_off$_match"; then
+		# off unless explicit match, but
+		# only turn it off if it was on
+		$DEBUG_DO _debugOff $_e $DEBUG_ON
+	fi
+	DEBUGGING=$DEBUG_SKIP	# backwards compatability
+	$DEBUG_DO set -x	# back on if needed
+	$DEBUG_DO set -x	# make sure we see it in trace
+	return $_rc
+}
+
+# Only turn debugging off if one of our args was the reason it
+# was turned on.
+# We normally return 0, but caller can pass rc=$? as first arg
+# so that we preserve the status of last statement.
+DebugOff() {
+	case ",${DEBUG_SH:-$DEBUG}," in
+	*,[Dd]ebug,*) ;;
+	*) $DEBUG_DO set +x;;		# reduce the noise
+	esac
+	_rc=0			# always happy
+	while :
+	do
+		case "$1" in
+		-[eo]) shift;;	# ignore it
+		rc=*) eval "_$1"; shift;;
+		*) break;;
+		esac
+	done
+	for _e in $*
+	do
+		: $_e==$DEBUG_OFF DEBUG_OFF
+		case "$DEBUG_OFF" in
+		"")	break;;
+		$_e)	_debugOn $DEBUG_ON; return $_rc;;
+		esac
+	done
+	for _e in $*
+	do
+		: $_e==$DEBUG_ON DEBUG_ON
+		case "$DEBUG_ON" in
+		"")	break;;
+		$_e)	_debugOff; return $_rc;;
+		esac
+	done
+	DEBUGGING=$DEBUG_SKIP	# backwards compatability
+	$DEBUG_DO set -x	# back on if needed
+	$DEBUG_DO set -x	# make sure we see it in trace
+	return $_rc
+}
+
+_TTY=${_TTY:-`test -t 0 && tty`}; export _TTY
+
+# override this if you like
+_debugShell() {
+	{
+		echo DebugShell "$@"
+		echo "Type 'exit' to continue..."
+	} > $_TTY
+	${DEBUG_SHELL:-${SHELL:-/bin/sh}} < $_TTY > $_TTY 2>&1
+}
+
+# Run an interactive shell if appropriate
+# Note: you can use $DEBUG_SKIP DebugShell ... to skip unless debugOn
+DebugShell() {
+	case "$_TTY%${DEBUG_INTERACTIVE}" in
+	*%|%*) return 0;;	# no tty or no spec
+	esac
+	for _e in ${*:-$Myname} all
+	do
+		case ",${DEBUG_INTERACTIVE}," in
+		*,!$_e,*|*,!$Myname:$_e,*)
+			return 0
+			;;
+		*,$_e,*|*,$Myname:$_e,*)
+			# Provide clues as to why/where
+			_debugShell "$_e: $@"
+			return $?
+			;;
+		esac
+	done
+	return 0
+}
+
+# For backwards compatability
+Debug() {
+	case "${DEBUG_SH:-$DEBUG}" in
+	"")	;;
+	*)	DEBUG_ON=${DEBUG_ON:-_Debug}
+		DebugOn -e $* || DebugOff $DEBUG_LAST
+		DEBUGGING=$DEBUG_SKIP
+		;;
+	esac
+}
diff --git a/libexec/rc/rc b/libexec/rc/rc
index 0ea61a4b2c0a..b23b0f35f263 100644
--- a/libexec/rc/rc
+++ b/libexec/rc/rc
@@ -66,8 +66,11 @@ fi
 # to minimize the number of files that are needed on a diskless system,
 # and to make the configuration file variables available to rc itself.
 #
+# -o verify has no effect if mac_veriexec is not active
+set -o verify
 . /etc/rc.subr
-load_rc_config
+set +o verify
+load_rc_config $rc_config_xtra
 
 # If we receive a SIGALRM, re-source /etc/rc.conf; this allows rc.d
 # scripts to perform "boot-time configuration" including enabling and
@@ -93,16 +96,7 @@ fi
 unset system_rc
 find_system_scripts
 files=`rcorder ${skip} ${skip_firstboot} ${system_rc} 2>/dev/null`
-
-_rc_elem_done=' '
-for _rc_elem in ${files}; do
-	run_rc_script ${_rc_elem} ${_boot}
-	_rc_elem_done="${_rc_elem_done}${_rc_elem} "
-
-	case "$_rc_elem" in
-	*/${early_late_divider})	break ;;
-	esac
-done
+run_rc_scripts --break ${early_late_divider} ${rc_early_flags} $files
 
 unset files local_rc system_rc
 
@@ -122,13 +116,13 @@ fi
 
 find_system_scripts
 files=`rcorder ${skip} ${skip_firstboot} ${system_rc} ${local_rc} 2>/dev/null`
-for _rc_elem in ${files}; do
-	case "$_rc_elem_done" in
-	*" $_rc_elem "*)	continue ;;
-	esac
+run_rc_scripts ${rc_late_flags} $files
+unset files local_rc system_rc
 
-	run_rc_script ${_rc_elem} ${_boot}
-done
+# allow for more complicated setups
+if have run_rc_scripts_final; then
+        run_rc_scripts_final
+fi
 
 # Remove the firstboot sentinel, and reboot if it was requested.
 # Be a bit paranoid about removing it to handle the common failure
diff --git a/libexec/rc/rc.subr b/libexec/rc/rc.subr
index 8cf812b06d45..19955fa83fbd 100644
--- a/libexec/rc/rc.subr
+++ b/libexec/rc/rc.subr
@@ -66,6 +66,122 @@ rc_service="$0"
 #	functions
 #	---------
 
+# is_verified file
+#	if VERIEXEC is active check that $file is verified
+#
+VERIEXEC="/sbin/veriexec"
+if test -x $VERIEXEC && $VERIEXEC -i active > /dev/null 2>&1; then
+	is_verified() { $VERIEXEC -x $1; }
+else
+	is_verified() { return 0; }
+fi
+
+# indicate that we have vdot
+_VDOT_SH=:
+
+# current state of O_VERIFY
+o_verify()
+{
+	set -o | sed -n '/^verify/s,.*[[:space:]],,p'
+}
+
+##
+# o_verify_set want [save]
+#
+# record current state of verify in $save
+# and set it to $want if different
+#
+o_verify_set() {
+	local x=$(o_verify)
+
+	[ -z "$x" ] && return 0
+	[ -z "$2" ] || eval $2=$x
+	[ "$x" = "$1" ] && return 0
+	case "$1" in
+	on)
+		set -o verify
+		;;
+	off)
+		set +o verify
+		;;
+	esac
+}
+
+# for unverified files
+dotted=
+dot()
+{
+	local f verify
+
+	o_verify_set off verify
+	for f in "$@"; do
+		if [ -f $f -a -s $f ]; then
+			dotted="$dotted $f"
+			. $f
+		fi
+	done
+	o_verify_set $verify
+}
+
+# try for verified, fallback to safe
+sdot()
+{
+	local f
+
+	for f in "$@"; do
+		[ -f $f -a -s $f ] || continue
+		vdot $f || safe_dot $f
+	done
+}
+
+# convenience function - skip if not verified
+vdot()
+{
+	local f rc=0 verify
+
+	o_verify_set on verify
+	for f in "$@"; do
+		[ -f $f -a -s $f ] || continue
+		if is_verified $f 2> /dev/null; then
+			dotted="$dotted $f"
+			. $f
+		else
+			rc=80	# EAUTH
+		fi
+	done
+	o_verify_set $verify
+	return $rc
+}
+
+# do we have $1 (could be a function)
+have()
+{
+       type "$1" > /dev/null 2>&1
+}
+
+# provide consistent means of logging progress
+rc_log()
+{
+	date "+@ %s [%Y-%m-%d %H:%M:%S %Z] $*"
+}
+
+# only rc_log if tracing enabled
+# and $level >= $RC_LEVEL
+rc_trace()
+{
+	local level=$1; shift
+	local cf=/etc/rc.conf.d/rc_trace
+
+	if [ -z "$RC_LEVEL" ]; then
+		[ -f $cf ] || return
+		[ -s $cf ] && \
+		RC_LEVEL=$(sed -n '/^RC_LEVEL=/ { s/.*=//p;q; }' $cf)
+		RC_LEVEL=${RC_LEVEL:-0}
+	fi
+	[ ${RC_LEVEL:-0} -ge ${level:-0} ] || return
+	rc_log "$@"
+}
+
 # list_vars pattern
 #	List variables matching glob pattern.
 #
@@ -924,6 +1040,8 @@ run_rc_command()
 		err 3 'run_rc_command: $name is not set.'
 	fi
 
+	DebugOn rc:$name rc:$name:$rc_arg $name:$rc_arg
+
 	# Don't repeat the first argument when passing additional command-
 	# line arguments to the command subroutines.
 	#
@@ -1077,6 +1195,7 @@ run_rc_command()
 		     _postcmd=\$${rc_arg}_postcmd
 
 		if [ -n "$_cmd" ]; then
+			rc_trace 1 "$_cmd"
 			if [ -n "$_env" ]; then
 				eval "export -- $_env"
 			fi
@@ -1449,6 +1568,10 @@ run_rc_script()
 		required_vars
 	eval unset ${_arg}_cmd ${_arg}_precmd ${_arg}_postcmd
 
+	rc_trace 0 "$_file $_arg"
+	# don't use it if we don't trust it
+	is_verified $_file || return
+
 	rc_service="$_file"
 	case "$_file" in
 	/etc/rc.d/*.sh)			# no longer allowed in the base
@@ -1459,6 +1582,8 @@ run_rc_script()
 		;;
 	*)				# run in subshell
 		if [ -x $_file ]; then
+			DebugOn $_file $_file:$_arg rc:${_file##*/} rc:${_file##*/}:$_arg ${_file##*/} ${_file##*/}:$_arg
+
 			if [ -n "$rc_boottrace" ]; then
 				boottrace_fn "$_file" "$_arg"
 			elif [ -n "$rc_fast_and_loose" ]; then
@@ -1469,11 +1594,65 @@ run_rc_script()
 				  trap "echo Script $_file running >&2" 29
 				  set $_arg; . $_file )
 			fi
+			DebugOff $_file $_file:$_arg rc:${_file##*/} rc:${_file##*/}:$_arg ${_file##*/} ${_file##*/}:$_arg
 		fi
 		;;
 	esac
 }
 
+#
+# run_rc_scripts [options] file [...]
+#
+# Call `run_rc_script' for each "file" unless already listed in
+# $_rc_elem_done.
+#
+# Options:
+#
+#	--arg "arg"
+#		Pass "arg" to `run_rc_script' default is $_boot.
+#
+#	--break "marker"
+#		If any "file" matches "marker" stop processing.
+#
+_rc_elem_done=
+run_rc_scripts()
+{
+	local _arg=${_boot}
+	local _rc_elem
+	local _rc_breaks=
+
+	while :; do
+		case "$1" in
+		--arg)
+                        _arg="$2"
+                        shift 2
+                        ;;
+		--break)
+                        _rc_breaks="$_rc_breaks $2"
+                        shift 2
+                        ;;
+		*)
+                        break
+                        ;;
+		esac
+	done
+	for _rc_elem in "$@"; do
+		: _rc_elem=$_rc_elem
+		case " $_rc_elem_done " in
+		*" $_rc_elem "*)
+                        continue
+                        ;;
+		esac
+		run_rc_script ${_rc_elem} ${_arg}
+		_rc_elem_done="$_rc_elem_done $_rc_elem"
+		case " $_rc_breaks " in
+		*" ${_rc_elem##*/} "*)
+                        break
+                        ;;
+		esac
+	done
+}
+
 boottrace_fn()
 {
 	local _file _arg
@@ -1502,19 +1681,42 @@ boottrace_sysctl()
 #
 load_rc_config()
 {
-	local _name _rcvar_val _var _defval _v _msg _new _d
+	local _name _rcvar_val _var _defval _v _msg _new _d _dot
 	_name=$1
+	_dot=${load_rc_config_reader:-dot}
+
+	case "$_dot" in
+	dot|[sv]dot)
+		;;
+	*)	warn "Ignoring invalid load_rc_config_reader"
+		_dot=dot
+		;;
+	esac
+	case "$1" in
+	-s|--safe)
+                _dot=sdot
+                _name=$2
+                shift
+                ;;
+	-v|--verify)
+                _dot=vdot
+                _name=$2
+                shift
+                ;;
+	esac
+
+	DebugOn rc:$_name $_name
 
 	if ${_rc_conf_loaded:-false}; then
 		:
 	else
 		if [ -r /etc/defaults/rc.conf ]; then
 			debug "Sourcing /etc/defaults/rc.conf"
-			. /etc/defaults/rc.conf
+			$_dot /etc/defaults/rc.conf
 			source_rc_confs
 		elif [ -r /etc/rc.conf ]; then
 			debug "Sourcing /etc/rc.conf (/etc/defaults/rc.conf doesn't exist)."
-			. /etc/rc.conf
+			$_dot /etc/rc.conf
 		fi
 		_rc_conf_loaded=true
 	fi
@@ -1526,13 +1728,13 @@ load_rc_config()
 			_d=${_d%/rc.d}
 			if [ -f ${_d}/rc.conf.d/"$_name" ]; then
 				debug "Sourcing ${_d}/rc.conf.d/$_name"
-				. ${_d}/rc.conf.d/"$_name"
+				$_dot ${_d}/rc.conf.d/"$_name"
 			elif [ -d ${_d}/rc.conf.d/"$_name" ] ; then
 				local _rc
 				for _rc in ${_d}/rc.conf.d/"$_name"/* ; do
 					if [ -f "$_rc" ] ; then
 						debug "Sourcing $_rc"
-						. "$_rc"
+						$_dot "$_rc"
 					fi
 				done
 			fi
@@ -2286,3 +2488,24 @@ boottrace_cmd=`command -v boottrace`
 if [ -n "$boottrace_cmd" ] && [ "`${SYSCTL_N} -q kern.boottrace.enabled`" = "1" ]; then
 	rc_boottrace=YES
 fi
+
+# Allow for local additions and overrides.
+# Use vdot to ensure the file has not been tampered with.
+vdot /etc/local.rc.subr
+
+# safe_eval.sh provides safe_dot - for untrusted files
+$_SAFE_EVAL_SH vdot /libexec/safe_eval.sh
+$_DEBUG_SH vdot /libexec/debug.sh
+
+# Ensure we can still operate if debug.sh and
+# safe_eval.sh are not found.
+if have DebugOn; then
+	# allow DEBUG_SH to be set from loader prompt
+	DEBUG_SH=${DEBUG_SH:-$(kenv -q DEBUG_SH)}
+else
+	DebugOn() { return 0; }
+	DebugOff() { return 0; }
+fi
+if ! have save_dot; then
+	safe_dot() { dot "$@"; }
+fi
diff --git a/libexec/rc/safe_eval.sh b/libexec/rc/safe_eval.sh
new file mode 100644
index 000000000000..bd9bc9394814
--- /dev/null
+++ b/libexec/rc/safe_eval.sh
@@ -0,0 +1,66 @@
+# SPDX-License-Identifier: BSD-2-Clause
+
+# RCSid:
+#	$Id: safe_eval.sh,v 1.12 2023/10/12 18:46:53 sjg Exp $
+#
+#	@(#) Copyright (c) 2023 Simon J. Gerraty
+#
+#	This file is provided in the hope that it will
+#	be of use.  There is absolutely NO WARRANTY.
+#	Permission to copy, redistribute or otherwise
+#	use this file is hereby granted provided that
+#	the above copyright notice and this notice are
+#	left intact.
+#
+#	Please send copies of changes and bug-fixes to:
+#	sjg@crufty.net
+
+_SAFE_EVAL_SH=:
+
+##
+# safe_set
+#
+# return a safe variable setting
+# any non-alphanumeric chars are replaced with '_'
+#
+safe_set() {
+    sed 's/[ 	]*#.*//;/^[A-Za-z_][A-Za-z0-9_]*=/!d;s;[^A-Za-z0-9_. 	"$,/=-];_;g'
+}
+
+##
+# safe_eval [file]
+#
+# eval variable assignments only from file
+# taking care to eliminate any shell meta chars
+#
+safe_eval() {
+    eval `cat "$@" | safe_set`
+}
+
+##
+# safe_dot file [...]
+#
+# feed all "file" that exist to safe_eval
+#
+safe_dot() {
+    local ef= f
+
+    for f in "$@"
+    do
+        test -s $f || continue
+        ef="${ef:+$ef }$f"
+        dotted="$dotted $f"
+    done
+    test -z "$ef" && return 1
+    safe_eval $ef
+    return 0
+}
+
+case /$0 in
+*/safe_eval*)
+    case "$1" in
+    dot|eval|set) op=safe_$1; shift; $op "$@";;
+    *) safe_dot "$@";;
+    esac
+    ;;
+esac
diff --git a/share/man/man8/Makefile b/share/man/man8/Makefile
index 1b942e275209..1e2c22e97d8e 100644
--- a/share/man/man8/Makefile
+++ b/share/man/man8/Makefile
@@ -4,6 +4,7 @@
 MAN=	\
 	beinstall.8 \
 	crash.8 \
+	debug.sh.8 \
 	diskless.8 \
 	intro.8 \
 	nanobsd.8 \
diff --git a/share/man/man8/debug.sh.8 b/share/man/man8/debug.sh.8
new file mode 100644
index 000000000000..2c137ff3fd42
--- /dev/null
+++ b/share/man/man8/debug.sh.8
@@ -0,0 +1,182 @@
+.\" Copyright (c) 1994-2021 Simon J. Gerraty
+.\"
+.\" SPDX-License-Identifier: BSD-2-Clause
+.\"
+.\" This file is provided in the hope that it will
+.\" be of use.  There is absolutely NO WARRANTY.
+.\" Permission to copy, redistribute or otherwise
+.\" use this file is hereby granted provided that
+.\" the above copyright notice and this notice are
+.\" left intact.
+.\"
+.\" Please send copies of changes and bug-fixes to:
+.\" sjg@crufty.net
+.\"
+.Dd January 31, 2024
+.Dt DEBUG.SH 8
+.Os
+.Sh NAME
+.Nm debug.sh
+.Nd selectively debug scripts
+.Sh SYNOPSIS
+.Bl -item -compact
+.It
+.Ic $_DEBUG_SH .\& Pa debug.sh
+.Pp
+.It
+.Ic DebugOn Oo Fl eo Oc Ar tag ...
+.It
+.Ic DebugOff Oo Fl eo Oc Oo Cm rc= Ns Ar rc Oc Ar tag ...
+.It
+.Ic Debugging
+.It
+.Ic DebugEcho Op Ar message
+.It
+.Ic DebugLog Op Ar message
+.It
+.Ic DebugShell Ar tag ...
+.It
+.Ic DebugTrace Ar message
+.It
+.Ic Debug Ar tag ...
+.El
+.Sh DESCRIPTION
+.Nm
+provides the following functions to facilitate flexible
+run-time tracing of complicated shell scripts.
+.Bl -tag -width 4n
+.It Ic DebugOn Oo Fl eo Oc Ar tag ...
+turns tracing on if any
+.Ar tag
+is found in
+.Va DEBUG_SH
+(a comma separated list of tags).
+.Pp
+It turns tracing off if
+.Ar !tag
+is found in
+.Va DEBUG_SH .
+.Pp
+It sets
+.Va DEBUG_ON
+to the
+.Ar tag
+that caused tracing to be enabled, or
+.Va DEBUG_OFF
+if we matched
+.Ar !tag .
+.Pp
+If
+.Fl e
+option is present, returns 1 if no
+.Ar tag
+matched.
+.Pp
+If
+.Fl o
+option is present, tracing is turned off unless there
+was a matched
+.Ar tag ,
+useful for functions too noisy to tace.
+.It Ic DebugOff Oo Fl eo Oc Oo Cm rc= Ns Ar rc Oc Ar tag ...
+turns tracing on if any
+.Ar tag
+matches
+.Va DEBUG_OFF
+or off if any
+.Ar tag
+matches
+.Va DEBUG_ON .
+This allows nested functions to not interfere with each other.
+.Pp
+The flags
+.Fl e
+and
+.Fl o
+are ignored, they just allow for symmetry with calls to
+.Fn DebugOn .
+.Pp
+The optional
+.Ar rc
+value will be returned rather than the default of 0.
+Thus if 
+.Fn DebugOff
+is the last operation in a function,
+.Ar rc
+will be the return code of the function.
+.It Ic Debugging
+returns true if tracing is enabled.
+It is useful for bounding complex debug actions, rather than
+using lots of
+.Ic $DEBUG_DO
+lines.
+.It Ic DebugEcho
+is just shorthand for:
+.Bd -literal -offset indent
+$DEBUG_DO echo "$@"
+.Ed
+.It Ic DebugLog Op Ar message
+If debugging is enabled, output
+.Ar message
+prefixed with a time-stamp.
+.It Ic DebugShell Ar tag ...
+runs an interactive shell if any
+.Ar tag
+is found in
+.Va DEBUG_INTERACTIVE ,
+and there is a tty available.
+The shell used is defined by
+.Va DEBUG_SHELL
+or
+.Va SHELL
+and defaults to
+.Pa /bin/sh .
+.It Ic DebugTrace Ar message
+Debug output can be very noisy, and it can be tricky
+to align with the script.
+This function outputs a very noticable banner indicating the value of
+.Va DEBUG_ON ,
+and
+.Ar message
+is passed to
+.Fn DebugLog ,
+finally the banner is repeated.
+.It Ic Debug Ar tag ...
+For backwards compatibility, calls
+.Fn DebugOn
+and if that does not turn tracing on,
+it calls
+.Fn DebugOff
+to turn it off.
+.El
+.Pp
+The variables
+.Va DEBUG_SKIP
+and
+.Va DEBUG_DO
+are set so as to enable/disable code that should be
+skipped/run when debugging is turned on.
+.Va DEBUGGING
+is the same as
+.Va DEBUG_SKIP
+for backwards compatability and is only set by
*** 396 LINES SKIPPED ***