From nobody Wed Oct 26 21:24:02 2022 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MyMJ329Kyz4g6XR; Wed, 26 Oct 2022 21:24:03 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4MyMJ31c3Hz4C9S; Wed, 26 Oct 2022 21:24:03 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1666819443; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=vS+ANNL0G5bAK9Csf7fBjEUqMEV1JQAnM6k01+lOrQA=; b=NGJ/84/s7OvOQrEgsKgfo5QoOQkcfIStcTiplkewGeZ7oYzFOwUo0IoQtyrpwZi8AThjJW tKal51Gtr/kcJUY+Y+nMOe/zK8x+UPqly7CC99HzTMWG+bcp2ZUm+D/mp/z9q73hqduxtS MRi1RXp9+RGqcQfx4uvFs09Ocw/+5KjjdVnk8d5myVh//zmTEjJJYHWQlcO7DGGiM2+ayo borvLWCJPNztAyUaIZSYbVHN6nViAxhiieEdUZOgkmD68wZTyprSNBgTAw83BMxyH3Sq7F CZen+PEMo42TggWCNDtXpMYldWoSKURyeftE3XDuxCnvKiZMgOJIvFko+zVpew== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4MyMJ30LZbzXST; Wed, 26 Oct 2022 21:24:03 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 29QLO25e080463; Wed, 26 Oct 2022 21:24:02 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 29QLO2Ud080462; Wed, 26 Oct 2022 21:24:02 GMT (envelope-from git) Date: Wed, 26 Oct 2022 21:24:02 GMT Message-Id: <202210262124.29QLO2Ud080462@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: John Baldwin Subject: git: 769b884e2e2e - main - vmm: Fix AP startup with old userspace binaries. List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: jhb X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 769b884e2e2eb84d25eca2a5462ae0a6c4dcd2a7 Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1666819443; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=vS+ANNL0G5bAK9Csf7fBjEUqMEV1JQAnM6k01+lOrQA=; b=jjUYXqrs4nQZpoOAQAPokHjXkkET4zk/ViZNgEfDguRLVJwh835h1tWYnOLKukYHcijyW0 LGRLUBQSVl9DpF8MpyuDH8c5MWKr0RLakxF8RB8vo4/6cTN7ti3+OeOzPfIX5Vnlm6SK97 KkBaic1OXFERCNxbHri1uTvVNlpTl0zBdJYXjX6ln9bgTUMbiptiyfuIKn6CQt6vZ6tOr8 1fwoSUge6Oauo9jgOiE8o+LrU8YzhXx+FAs6KIFHZ/rbkyBJZq57mfb9WkDRS61F5jNYxf NvcmvYA4KgoLbn4WYpfGKLgBr3R2mIIrqyDmSSbYOuFQvIXHSo5UGagfsvN6/g== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1666819443; a=rsa-sha256; cv=none; b=MaPSqcxM7AgK5ua98qu8z7ESxGBw+nB63BrMn6u0Dz8hSJVpzPlWazG5Dq/RjZzjDV1Qpa JzjjLbUuNzP0ghebhriY400ZhxwsJlIz0vfh0YBvGCx8q5kkwGK4ujm960kkl/TdWxn6FO ZKyqBFS/fScoVeZfvhB7mMWJbXGplzU0l4ja1iTk1AczHbTTgix2olJgctNVFo7TFsTlIv OlE5JcgjAr5Y9+a7s4/SJ+3nwCgn6kE2uqK9t+3YYlaLUepdrH8Fl8BgTwSIb58SAFTpfz mkYod3almaSQwnHuQkDbqmJWYUytt0ulrsHgOG9q/GKy4XPcPj050AuZSHrnwg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by jhb: URL: https://cgit.FreeBSD.org/src/commit/?id=769b884e2e2eb84d25eca2a5462ae0a6c4dcd2a7 commit 769b884e2e2eb84d25eca2a5462ae0a6c4dcd2a7 Author: John Baldwin AuthorDate: 2022-10-26 21:22:56 +0000 Commit: John Baldwin CommitDate: 2022-10-26 21:22:56 +0000 vmm: Fix AP startup with old userspace binaries. Older binaries that do not request IPI exits to userspace do not start user threads for other vCPUs until a STARTUP IPI triggers a VM_EXITCODE_SPINUP_AP exit to userland. This means that those vcpus are not yet active (in terms of vm_active_cpus) when the INIT and STARTUP IPIs are delivered to the vCPUs. The changes in commit 0bda8d3e9f7a changed the INIT and STARTUP IPIs to reuse the existing vlapic_calcdest() function. This function silently ignores IPIs sent to inactive vCPUs. As a result, when using an old bhyve binary, the INIT and STARTUP IPIs sent to wakeup APs were ignored. To fix, restructure the compat code for the INIT and STARTUP IPIs to ignore the results of vlapic_calcdest() and manually parse the APIC ID and resulting vcpuid. As part of this, make the compat code always conditonal on the ipi_exit capability being disabled. Reviewed by: c.koehne_beckhoff.com, markj Differential Revision: https://reviews.freebsd.org/D37093 --- sys/amd64/vmm/io/vlapic.c | 63 +++++++++++++++++++++++++++++++++-------------- 1 file changed, 45 insertions(+), 18 deletions(-) diff --git a/sys/amd64/vmm/io/vlapic.c b/sys/amd64/vmm/io/vlapic.c index 8b547d9a9453..0e1fb1bbf006 100644 --- a/sys/amd64/vmm/io/vlapic.c +++ b/sys/amd64/vmm/io/vlapic.c @@ -1119,20 +1119,61 @@ vlapic_icrlo_write_handler(struct vlapic *vlapic, bool *retu) break; case APIC_DELMODE_INIT: - CPU_FOREACH_ISSET(i, &dmask) { + if (!vlapic->ipi_exit) { + if (!phys) + break; + + i = vm_apicid2vcpuid(vlapic->vm, dest); + if (i >= vm_get_maxcpus(vlapic->vm) || + i == vlapic->vcpuid) + break; + /* - * Userland which doesn't support the IPI exit requires - * that the boot state is set to SIPI here. + * Userland which doesn't support the IPI exit + * requires that the boot state is set to SIPI + * here. */ vlapic2 = vm_lapic(vlapic->vm, i); vlapic2->boot_state = BS_SIPI; - CPU_SET(i, &ipimask); + break; } + CPU_COPY(&dmask, &ipimask); break; case APIC_DELMODE_STARTUP: + if (!vlapic->ipi_exit) { + if (!phys) + break; + + /* + * Old bhyve versions don't support the IPI + * exit. Translate it into the old style. + */ + i = vm_apicid2vcpuid(vlapic->vm, dest); + if (i >= vm_get_maxcpus(vlapic->vm) || + i == vlapic->vcpuid) + break; + + /* + * Ignore SIPIs in any state other than wait-for-SIPI + */ + vlapic2 = vm_lapic(vlapic->vm, i); + if (vlapic2->boot_state != BS_SIPI) + break; + vlapic2->boot_state = BS_RUNNING; + + vmexit = vm_exitinfo(vlapic->vm, vlapic->vcpuid); + vmexit->exitcode = VM_EXITCODE_SPINUP_AP; + vmexit->u.spinup_ap.vcpu = i; + vmexit->u.spinup_ap.rip = vec << PAGE_SHIFT; + + *retu = true; + break; + } + CPU_FOREACH_ISSET(i, &dmask) { vlapic2 = vm_lapic(vlapic->vm, i); + /* * Ignore SIPIs in any state other than wait-for-SIPI */ @@ -1155,20 +1196,6 @@ vlapic_icrlo_write_handler(struct vlapic *vlapic, bool *retu) vmexit->u.ipi.dmask = dmask; *retu = true; - - /* - * Old bhyve versions don't support the IPI exit. Translate it - * into the old style. - */ - if (!vlapic->ipi_exit) { - if (mode == APIC_DELMODE_STARTUP) { - vmexit->exitcode = VM_EXITCODE_SPINUP_AP; - vmexit->u.spinup_ap.vcpu = CPU_FFS(&ipimask) - 1; - vmexit->u.spinup_ap.rip = vec << PAGE_SHIFT; - } else { - *retu = false; - } - } } return (0);