From nobody Mon Jan 17 10:09:28 2022 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 7041B194AF87; Mon, 17 Jan 2022 10:09:28 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Jcngr2czxz3qmk; Mon, 17 Jan 2022 10:09:28 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1642414168; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=UuLTBUNjKeVh54F+0rVLGT+eT/fdWwvEYcToIppahrw=; b=ZZtt9C73Y4fnXaro5aRDbCMOKO6UA5NUdtuM5mdwr9V2AoN1+VOsvgMp1VFEj+Pww+q2N1 gPVYwCQnqMVPnosJaI6CPMOVO6HEBDQgkvxASrscePUGcdiGUOqC8nkz6G2csoUDNS20jJ wPqSDXzK/8AmTt5nCXobTg5yB4xh3yveOzZGi9jaZ5GlRrcjlP+0hU33gnxcYhk9uBCmQ+ qNR8x3+GVqxUy67S7EClbkC9bZvyMRHJVP+UAUQKYzWFfkGw2OziQN07KWzlcyzx/UbGK0 eKECu2A+yXaI9N2vAl1wrDaL63nemIFKfT9QxEt+71Dqxu9DXHa16zxhV5uIvA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 3746A3AE3; Mon, 17 Jan 2022 10:09:28 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 20HA9SLj010628; Mon, 17 Jan 2022 10:09:28 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 20HA9S32010627; Mon, 17 Jan 2022 10:09:28 GMT (envelope-from git) Date: Mon, 17 Jan 2022 10:09:28 GMT Message-Id: <202201171009.20HA9S32010627@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: =?utf-8?Q?Roger Pau Monn=C3=A9?= Subject: git: ad15eeeaba30 - main - x86/xen: fallback when VCPUOP_send_nmi is not available List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: royger X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: ad15eeeaba30cdf10036b7869d27441cfc9f0674 Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1642414168; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=UuLTBUNjKeVh54F+0rVLGT+eT/fdWwvEYcToIppahrw=; b=v+bKp64BJKRo+8RFZek4M93kr+YARAgFTla0w7c/cKG3tsz7sb49T0b8xgsfSxGTsSHmUB 7SxgnZ0hUdi+7uH4mQj3XmLnewH7QoBsFBLv7wMe+2R+iSqpk1Lqyz4TAEGNt+VZ9LHJEK r5DiWz/WRcftOAd2GOi5Fs0985in3rwHLWI5+/IseuTkhiOkAFGd7sWk4ekh147131E75U 8xPnVvb+N/fzw/vHqvov7MpLIQLripbiHL19QrqiUk1JSYF4UU0Jrfkb+SmcW2PFs7Fq5T 2PNJvPCa8wvQAIu0saAkQgVTyPMLHK5fUtlGba0ASrvuy8bSOXtWdZ8DERaycg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1642414168; a=rsa-sha256; cv=none; b=CpDt03cSsBRmXJ72PuQvUzhawQ+BXwZBP6RxUwPOpjoFnXgNfmMzEX8vnzsJeWyl4aRIyF mxXc4LRxw59gxmUGuzlIDI2uc2v9muuUV2voesbloeW8igMAJiLVZ1wumKPZ8fjXtlxQwB Ys5SSRQewX7CXDfiCUa8CTBh/5aY1+SzDAMWuvknUV6jWxTZuq4UCAu6moUlej8Ck2gZrJ pvhjSVr9MVlJ7PEWYgkzpu9BtQhuqsVr5ZtcklJsVQNAluMexPdRKEeHIkYMFJs82Jyuev SswXYMKwf4Q1ufPqz2jv+3F2SHWXTxhv8plB9ujgJAkhHZUqE2WpmyTFy51NLw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by royger: URL: https://cgit.FreeBSD.org/src/commit/?id=ad15eeeaba30cdf10036b7869d27441cfc9f0674 commit ad15eeeaba30cdf10036b7869d27441cfc9f0674 Author: Roger Pau Monné AuthorDate: 2022-01-13 13:48:14 +0000 Commit: Roger Pau Monné CommitDate: 2022-01-17 10:06:40 +0000 x86/xen: fallback when VCPUOP_send_nmi is not available It has been reported that on some AWS instances VCPUOP_send_nmi returns -38 (ENOSYS). The hypercall is only available for HVM guests in Xen 4.7 and newer. Add a fallback to use the native NMI sending procedure when VCPUOP_send_nmi is not available, so that the NMI is not lost. Reported and Tested by: avg MFC after: 1 week Fixes: b2802351c162 ('xen: fix dispatching of NMIs') Sponsored by: Citrix Systems R&D --- sys/x86/xen/xen_apic.c | 52 +++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 41 insertions(+), 11 deletions(-) diff --git a/sys/x86/xen/xen_apic.c b/sys/x86/xen/xen_apic.c index b553e5248716..2efa5a3c2345 100644 --- a/sys/x86/xen/xen_apic.c +++ b/sys/x86/xen/xen_apic.c @@ -107,6 +107,14 @@ static struct xen_ipi_handler xen_ipis[] = }; #endif +/* + * Save previous (native) handler as a fallback. Xen < 4.7 doesn't support + * VCPUOP_send_nmi for HVM guests, and thus we need a fallback in that case: + * + * https://lists.freebsd.org/archives/freebsd-xen/2022-January/000032.html + */ +void (*native_ipi_vectored)(u_int, int); + /*------------------------------- Per-CPU Data -------------------------------*/ #ifdef SMP DPCPU_DEFINE(xen_intr_handle_t, ipi_handle[nitems(xen_ipis)]); @@ -273,10 +281,11 @@ xen_pv_lapic_ipi_raw(register_t icrlo, u_int dest) } #define PCPU_ID_GET(id, field) (pcpu_find(id)->pc_##field) -static void +static int send_nmi(int dest) { unsigned int cpu; + int rc = 0; /* * NMIs are not routed over event channels, and instead delivered as on @@ -286,24 +295,33 @@ send_nmi(int dest) */ switch(dest) { case APIC_IPI_DEST_SELF: - HYPERVISOR_vcpu_op(VCPUOP_send_nmi, PCPU_GET(vcpu_id), NULL); + rc = HYPERVISOR_vcpu_op(VCPUOP_send_nmi, PCPU_GET(vcpu_id), NULL); break; case APIC_IPI_DEST_ALL: - CPU_FOREACH(cpu) - HYPERVISOR_vcpu_op(VCPUOP_send_nmi, + CPU_FOREACH(cpu) { + rc = HYPERVISOR_vcpu_op(VCPUOP_send_nmi, PCPU_ID_GET(cpu, vcpu_id), NULL); + if (rc != 0) + break; + } break; case APIC_IPI_DEST_OTHERS: - CPU_FOREACH(cpu) - if (cpu != PCPU_GET(cpuid)) - HYPERVISOR_vcpu_op(VCPUOP_send_nmi, + CPU_FOREACH(cpu) { + if (cpu != PCPU_GET(cpuid)) { + rc = HYPERVISOR_vcpu_op(VCPUOP_send_nmi, PCPU_ID_GET(cpu, vcpu_id), NULL); + if (rc != 0) + break; + } + } break; default: - HYPERVISOR_vcpu_op(VCPUOP_send_nmi, + rc = HYPERVISOR_vcpu_op(VCPUOP_send_nmi, PCPU_ID_GET(apic_cpuid(dest), vcpu_id), NULL); break; } + + return rc; } #undef PCPU_ID_GET @@ -312,9 +330,21 @@ xen_pv_lapic_ipi_vectored(u_int vector, int dest) { xen_intr_handle_t *ipi_handle; int ipi_idx, to_cpu, self; + static bool pvnmi = true; if (vector >= IPI_NMI_FIRST) { - send_nmi(dest); + if (pvnmi) { + int rc = send_nmi(dest); + + if (rc != 0) { + printf( + "Sending NMI using hypercall failed (%d) switching to APIC\n", rc); + pvnmi = false; + native_ipi_vectored(vector, dest); + } + } else + native_ipi_vectored(vector, dest); + return; } @@ -580,8 +610,8 @@ xen_setup_cpus(void) xen_cpu_ipi_init(i); /* Set the xen pv ipi ops to replace the native ones */ - if (xen_hvm_domain()) - apic_ops.ipi_vectored = xen_pv_lapic_ipi_vectored; + native_ipi_vectored = apic_ops.ipi_vectored; + apic_ops.ipi_vectored = xen_pv_lapic_ipi_vectored; } /* Switch to using PV IPIs as soon as the vcpu_id is set. */