From nobody Wed Sep 10 16:00:55 2025 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cMQQc15vGz66WLp; Wed, 10 Sep 2025 16:00:56 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R12" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cMQQb1h3qz3KMt; Wed, 10 Sep 2025 16:00:55 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1757520055; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=FdEnlTtAv0tsKewz4Nv/xfXWQiPlv7uMH9jqsBhThlk=; b=QtejRq2JYY1edaOnvLF8K2JcwedFOiwoTBnVrjwpCmW1/LZlWbXj30Xk3qDqMt5p6AECKw yIvDehBND84B7oTmCGkWGu9VO5t2tElNvGaUsVeu0MAFEtYQxu7I/ZbUKMuYibPHIUEuvx ugG8sasNsmygrMLNX/LCSNhgoiHtRMPaDqJ63ajGtX61GHzizKGCQfGxu5QOu2MHOOSGa3 aMuHR/euDT3igaP//UtCiW8xNdktbVtmqAWQyH/5cvP0m1iB3/wfjkYIFZJWdFS47qP8fl 7pCkTGTLbDv/t123UxESPaDDFjV4yg8+4bBimko2bIFuKFKqeCItklkg8iFRkQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1757520055; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=FdEnlTtAv0tsKewz4Nv/xfXWQiPlv7uMH9jqsBhThlk=; b=atSfpQtm1Vylj1aFmtdhijvd6X6F3EXZGdbP2ShuqyCU28xFskthXChwSj0VSyKSGRnvU3 MVbCaxFM43hR4IoJj4sfB7ZJZpScynQI1KNFsciIPjHWq73WO0FfebpPTwG3+ruMBFT3Q0 LZ0/xbzAXUkkNgYUScBCb/ceBTO4FCZ4IfXgQgqzC4qv1kcFrWgeoFoxUq8anIpDut9gsl DkxdHL4nfdBk6bzoJ7uwTtAVEkS5ovkz78vDsg3GmGhMaotWNj3IH1PD/sNmL4pYKOTO9D qEFfPgHzJnhwSW0Cv6EDcT7wyvKNwIDcHBND1kpKTFVBQ3G0I1O77lqqtGl/AA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1757520055; a=rsa-sha256; cv=none; b=TCgnsmCo1YtWzNtIQmry9CWVt6bVZENkvtvby28wZMZW9+hmxxgN/CCb19+SAJnJGdCy0l ZzeeC4W9LyRhgJpv/I02WnPYS++VgjyzpHeXbAA0+AQUFZIlXJHgHR86ZIRfQxmu3/DDem lX76LJhWeBViCZHIiXNeMfg+J2BTcFyhJQF9MmkiVWAHzIJ9ilGAs7tTmayVJPj7aWem38 nTWoM3QDt//dhx6pKTg8WzrGgZPfYa2y20lANynYKqR7YKMKokRnhbXipKD5AQuWo0xbTk r+KyxFHr+ArTpwjRnxgQP0N8FVGoB0RZwiivO54vbJLNqMgs1my4Gqt6RQFgLw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4cMQQb1DxhzC5L; Wed, 10 Sep 2025 16:00:55 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 58AG0tEh058520; Wed, 10 Sep 2025 16:00:55 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 58AG0tld058513; Wed, 10 Sep 2025 16:00:55 GMT (envelope-from git) Date: Wed, 10 Sep 2025 16:00:55 GMT Message-Id: <202509101600.58AG0tld058513@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Mark Johnston Subject: git: 3d39856d4dfe - main - vmm: Suspend the VM before destroying it List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: markj X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 3d39856d4dfeab5b5a5e6bbdb6ce965db5bc4dc1 Auto-Submitted: auto-generated The branch main has been updated by markj: URL: https://cgit.FreeBSD.org/src/commit/?id=3d39856d4dfeab5b5a5e6bbdb6ce965db5bc4dc1 commit 3d39856d4dfeab5b5a5e6bbdb6ce965db5bc4dc1 Author: Mark Johnston AuthorDate: 2025-09-10 16:00:36 +0000 Commit: Mark Johnston CommitDate: 2025-09-10 16:00:36 +0000 vmm: Suspend the VM before destroying it Otherwise we don't do anything to kick vcpu threads out of a sleep state when destroying a VM. For instance, suppose a guest executes hlt on amd64 or wfi on arm64 with interrupts disabled. Then, bhyvectl --destroy will hang until the vcpu thread somehow comes out of vm_handle_hlt()/vm_handle_wfi() since destroy_dev() is waiting for vCPU threads to drain. Note that on amd64, if hw.vmm.halt_detection is set to 1 (the default), the guest will automatically exit in this case since it's treated as a shutdown. But, the above should not hang if halt_detection is set to 0. Here, vm_suspend() wakes up vcpu threads, and a subsequent attempt to run the vCPU will result in an error which gets propagated to userspace, allowing destroy_dev() to proceed. Add a new suspend code for this purpose. Modify bhyve to exit with status 4 ("exited due to an error") when it's received, since that's what'll happen generally when the VM is destroyed asynchronously. Reported by: def MFC after: 2 weeks Sponsored by: Innovate UK Differential Revision: https://reviews.freebsd.org/D51761 --- sys/amd64/include/vmm.h | 1 + sys/arm64/include/vmm.h | 1 + sys/arm64/vmm/vmm.c | 6 ++++++ sys/dev/vmm/vmm_dev.c | 1 + sys/riscv/include/vmm.h | 1 + sys/riscv/vmm/vmm.c | 6 +++++- usr.sbin/bhyve/aarch64/vmexit.c | 2 ++ usr.sbin/bhyve/amd64/vmexit.c | 2 ++ usr.sbin/bhyve/bhyve.8 | 2 +- usr.sbin/bhyve/bhyverun.c | 6 ++---- usr.sbin/bhyve/riscv/vmexit.c | 2 ++ 11 files changed, 24 insertions(+), 6 deletions(-) diff --git a/sys/amd64/include/vmm.h b/sys/amd64/include/vmm.h index 0b3daed4f69e..e35119af8572 100644 --- a/sys/amd64/include/vmm.h +++ b/sys/amd64/include/vmm.h @@ -46,6 +46,7 @@ enum vm_suspend_how { VM_SUSPEND_POWEROFF, VM_SUSPEND_HALT, VM_SUSPEND_TRIPLEFAULT, + VM_SUSPEND_DESTROY, VM_SUSPEND_LAST }; diff --git a/sys/arm64/include/vmm.h b/sys/arm64/include/vmm.h index 73b5b4a09591..e839b5dd92c9 100644 --- a/sys/arm64/include/vmm.h +++ b/sys/arm64/include/vmm.h @@ -42,6 +42,7 @@ enum vm_suspend_how { VM_SUSPEND_RESET, VM_SUSPEND_POWEROFF, VM_SUSPEND_HALT, + VM_SUSPEND_DESTROY, VM_SUSPEND_LAST }; diff --git a/sys/arm64/vmm/vmm.c b/sys/arm64/vmm/vmm.c index 3082d2941221..1dcefa1489e9 100644 --- a/sys/arm64/vmm/vmm.c +++ b/sys/arm64/vmm/vmm.c @@ -1342,8 +1342,14 @@ vm_handle_smccc_call(struct vcpu *vcpu, struct vm_exit *vme, bool *retu) static int vm_handle_wfi(struct vcpu *vcpu, struct vm_exit *vme, bool *retu) { + struct vm *vm; + + vm = vcpu->vm; vcpu_lock(vcpu); while (1) { + if (vm->suspend) + break; + if (vgic_has_pending_irq(vcpu->cookie)) break; diff --git a/sys/dev/vmm/vmm_dev.c b/sys/dev/vmm/vmm_dev.c index 9f2b009d02ec..460a508a60dc 100644 --- a/sys/dev/vmm/vmm_dev.c +++ b/sys/dev/vmm/vmm_dev.c @@ -901,6 +901,7 @@ vmmdev_lookup_and_destroy(const char *name, struct ucred *cred) sc->cdev = NULL; sx_xunlock(&vmmdev_mtx); + vm_suspend(sc->vm, VM_SUSPEND_DESTROY); destroy_dev(cdev); vmmdev_destroy(sc); diff --git a/sys/riscv/include/vmm.h b/sys/riscv/include/vmm.h index 1221521be368..de7119dd534a 100644 --- a/sys/riscv/include/vmm.h +++ b/sys/riscv/include/vmm.h @@ -49,6 +49,7 @@ enum vm_suspend_how { VM_SUSPEND_RESET, VM_SUSPEND_POWEROFF, VM_SUSPEND_HALT, + VM_SUSPEND_DESTROY, VM_SUSPEND_LAST }; diff --git a/sys/riscv/vmm/vmm.c b/sys/riscv/vmm/vmm.c index 7528ef6e4698..ec4514f70fa6 100644 --- a/sys/riscv/vmm/vmm.c +++ b/sys/riscv/vmm/vmm.c @@ -1036,10 +1036,14 @@ vm_raise_msi(struct vm *vm, uint64_t msg, uint64_t addr, int bus, int slot, static int vm_handle_wfi(struct vcpu *vcpu, struct vm_exit *vme, bool *retu) { + struct vm *vm; + vm = vcpu->vm; vcpu_lock(vcpu); - while (1) { + if (vm->suspend) + break; + if (aplic_check_pending(vcpu->cookie)) break; diff --git a/usr.sbin/bhyve/aarch64/vmexit.c b/usr.sbin/bhyve/aarch64/vmexit.c index 3acad4020a3c..2457cbe76b5e 100644 --- a/usr.sbin/bhyve/aarch64/vmexit.c +++ b/usr.sbin/bhyve/aarch64/vmexit.c @@ -122,6 +122,8 @@ vmexit_suspend(struct vmctx *ctx, struct vcpu *vcpu, struct vm_run *vmrun) exit(1); case VM_SUSPEND_HALT: exit(2); + case VM_SUSPEND_DESTROY: + exit(4); default: fprintf(stderr, "vmexit_suspend: invalid reason %d\n", how); exit(100); diff --git a/usr.sbin/bhyve/amd64/vmexit.c b/usr.sbin/bhyve/amd64/vmexit.c index 944f5de34645..14f89563fd0f 100644 --- a/usr.sbin/bhyve/amd64/vmexit.c +++ b/usr.sbin/bhyve/amd64/vmexit.c @@ -418,6 +418,8 @@ vmexit_suspend(struct vmctx *ctx, struct vcpu *vcpu, struct vm_run *vmrun) exit(2); case VM_SUSPEND_TRIPLEFAULT: exit(3); + case VM_SUSPEND_DESTROY: + exit(4); default: EPRINTLN("vmexit_suspend: invalid reason %d", how); exit(100); diff --git a/usr.sbin/bhyve/bhyve.8 b/usr.sbin/bhyve/bhyve.8 index 89c0b23961a8..c902c265da9e 100644 --- a/usr.sbin/bhyve/bhyve.8 +++ b/usr.sbin/bhyve/bhyve.8 @@ -1126,7 +1126,7 @@ powered off .It 2 halted .It 3 -triple fault +triple fault (amd64 only) .It 4 exited due to an error .El diff --git a/usr.sbin/bhyve/bhyverun.c b/usr.sbin/bhyve/bhyverun.c index 9ead49582a7d..bfc0b949a75d 100644 --- a/usr.sbin/bhyve/bhyverun.c +++ b/usr.sbin/bhyve/bhyverun.c @@ -561,10 +561,8 @@ fbsdrun_start_thread(void *param) #endif vm_loop(vi->ctx, vi->vcpu); - - /* not reached */ - exit(1); - return (NULL); + /* We get here if the VM was destroyed asynchronously. */ + exit(4); } void diff --git a/usr.sbin/bhyve/riscv/vmexit.c b/usr.sbin/bhyve/riscv/vmexit.c index 3bc83b3bef4e..985f8e4e9065 100644 --- a/usr.sbin/bhyve/riscv/vmexit.c +++ b/usr.sbin/bhyve/riscv/vmexit.c @@ -121,6 +121,8 @@ vmexit_suspend(struct vmctx *ctx, struct vcpu *vcpu, struct vm_run *vmrun) exit(1); case VM_SUSPEND_HALT: exit(2); + case VM_SUSPEND_DESTROY: + exit(4); default: fprintf(stderr, "vmexit_suspend: invalid reason %d\n", how); exit(100);