git: a9e4753b4fe2 - main - bhyve: Optionally put vCPUs back in the debug state after resuming

The branch main has been updated by markj:

URL: https://cgit.FreeBSD.org/src/commit/?id=a9e4753b4fe20d81263f61c4b7e4383739924898

commit a9e4753b4fe20d81263f61c4b7e4383739924898
Author:     Mark Johnston <markj@FreeBSD.org>
AuthorDate: 2024-08-07 19:27:23 +0000
Commit:     Mark Johnston <markj@FreeBSD.org>
CommitDate: 2024-08-07 19:27:23 +0000

    bhyve: Optionally put vCPUs back in the debug state after resuming
    
    When the gdb stub is configured to pause guest execution upon boot (i.e.,
    the "w" flag is passed to -G), vCPUs end up suspended in two senses: first,
    suspended by the GDB stub (marked in the vcpus_suspended set), and suspended
    by the kernel (because fbsdrun_addcpu() suspends APs before spawning their
    vCPU threads).  When the guest is resumed by the debugger, vCPUs are
    unsuspended in both senses, but this is not correct for APs.
    
    Hack around this problem by re-suspending vCPUs after the debugger
    resumes guest execution, if they were suspended beforehand.
    
    Reviewed by:    corvink, jhb
    MFC after:      2 weeks
    Sponsored by:   Innovate UK
    Differential Revision:  https://reviews.freebsd.org/D46196
---
 usr.sbin/bhyve/gdb.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/usr.sbin/bhyve/gdb.c b/usr.sbin/bhyve/gdb.c
index 7c04b0017c80..983e7deb61c9 100644
--- a/usr.sbin/bhyve/gdb.c
+++ b/usr.sbin/bhyve/gdb.c
@@ -938,8 +938,28 @@ gdb_cpu_add(struct vcpu *vcpu)
 	 * executing the first instruction.
 	 */
 	if (!CPU_EMPTY(&vcpus_suspended)) {
+		cpuset_t suspended;
+		int error;
+
+		error = vm_debug_cpus(ctx, &suspended);
+		assert(error == 0);
+
 		CPU_SET(vcpuid, &vcpus_suspended);
 		_gdb_cpu_suspend(vcpu, false);
+
+		/*
+		 * In general, APs are started in a suspended mode such that
+		 * they exit with VM_EXITCODE_DEBUG until the BSP starts them.
+		 * In particular, this refers to the kernel's view of the vCPU
+		 * state rather than our own.  If the debugger resumes guest
+		 * execution, vCPUs will be unsuspended from the kernel's point
+		 * of view, so we should restore the previous state before
+		 * continuing.
+		 */
+		if (CPU_ISSET(vcpuid, &suspended)) {
+			error = vm_suspend_cpu(vcpu);
+			assert(error == 0);
+		}
 	}
 	pthread_mutex_unlock(&gdb_lock);
 }