From nobody Tue Apr 12 08:07:06 2022 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id A30E011D3C05; Tue, 12 Apr 2022 08:07:06 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KcyxQ37xdz3GmV; Tue, 12 Apr 2022 08:07:06 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1649750826; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=mzXy1N2D84cCrOiOx3mukqU4pAa1lydvQ9ph3Rdz2eE=; b=e84WIargTxLTVRI/Rgj0tBxIhWYJvXlHAO5k9UKwskE/oqKOx7VwJuAwxIvMDlTO642dsO bd+fCyhYTX7RUXEDnnZbYdnU77VjwzVtPcSkMJEr04RbaWkNIfo3XBNeZ3w4tA9w/7dQoQ V/JOvMX9TYlJFoqONxo5gHz2IZtW40wchVC6U6S36+QflgYQpdMbHtZ2d0uxS03keB2qtH bb8YbpoOxv1uezquJOcHWzV1jtz7tsrsRc1CwTIN9K3OIGcHxWrt1Wc38WOGA39SasSxnm baUxOmx+6HgS961TvLO3h2wQBdXP4UKxzqWFC2djVd26wRdiaZa6xhJLyk+6rA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 3881E13D2E; Tue, 12 Apr 2022 08:07:06 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 23C876YY012823; Tue, 12 Apr 2022 08:07:06 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 23C8761C012822; Tue, 12 Apr 2022 08:07:06 GMT (envelope-from git) Date: Tue, 12 Apr 2022 08:07:06 GMT Message-Id: <202204120807.23C8761C012822@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: =?utf-8?Q?Roger Pau Monn=C3=A9?= Subject: git: a1a5c5351659 - stable/13 - x86/xen: fallback when VCPUOP_send_nmi is not available List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-branches@freebsd.org X-BeenThere: dev-commits-src-branches@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: royger X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: a1a5c53516598ee0cd9910fff7733ec4746c1bdb Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1649750826; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=mzXy1N2D84cCrOiOx3mukqU4pAa1lydvQ9ph3Rdz2eE=; b=tBWSnfeEiZ1/esuU5+Ww4nABCUky1rsb58g0HEVptkVkhQaU2IwSh6c+5JRk8G+69vNmgB DsIoIVsaY4JPiY9aRTU/aHg4BCtOMXbNHqwTkrTK0kIk/FCQBUDCoOzjnLHqI3N93mP3ol UolFxQZXCVLzx8EFXYfezweD3pj2H9qwoYwUyzGnriaov0QWwhsC0dMxE6VWqJwMLzOcaD +lNcrGuQAWFdVoI1PEcz2q43pltk1NLxy4oqI923aUv8/lL/NA2H/glPGSM7+ydWY/d1od a6W301d5DOnFrXfd9VspcByT5hzv3HRmeC+CCm3flvf08+QPYOUGBsY+Vpvt+w== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1649750826; a=rsa-sha256; cv=none; b=JY8FOQa0/7pObdNc0jDbmz49xFvfeALQ9cbOd15bAtuuF6JemTw7BmahT6PrUB92Y4ylFe 1Y8rOk0Za9mTB1ihMthuzEayNMK3iO8Oo4EILy3niFz/4IOxbnU9S30T9oi/jkBfdsKiYL OA1UzmNf7angbtOwwRapKK9o3C90rrv5BUwm/4a4K1yMVoVfH6ruDZb5V4dT6JqhGxr9Ur TXceLwWSouW/an2Zwubxwn5v2MDcArvfa/UtHYzfwPv/iUpKCzy7DFeExNazLJo2DC5cv/ 4BwNAqGe3xY5W1hbulC+x3rbQli2TMIF4yM8cd1XcNEg5pmydqyqSNa3xf6ySQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch stable/13 has been updated by royger: URL: https://cgit.FreeBSD.org/src/commit/?id=a1a5c53516598ee0cd9910fff7733ec4746c1bdb commit a1a5c53516598ee0cd9910fff7733ec4746c1bdb Author: Roger Pau Monné AuthorDate: 2022-01-13 13:48:14 +0000 Commit: Roger Pau Monné CommitDate: 2022-04-12 08:05:47 +0000 x86/xen: fallback when VCPUOP_send_nmi is not available It has been reported that on some AWS instances VCPUOP_send_nmi returns -38 (ENOSYS). The hypercall is only available for HVM guests in Xen 4.7 and newer. Add a fallback to use the native NMI sending procedure when VCPUOP_send_nmi is not available, so that the NMI is not lost. Reported and Tested by: avg Fixes: b2802351c162 ('xen: fix dispatching of NMIs') Sponsored by: Citrix Systems R&D (cherry picked from commit ad15eeeaba30cdf10036b7869d27441cfc9f0674) --- sys/x86/xen/xen_apic.c | 52 +++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 41 insertions(+), 11 deletions(-) diff --git a/sys/x86/xen/xen_apic.c b/sys/x86/xen/xen_apic.c index b553e5248716..2efa5a3c2345 100644 --- a/sys/x86/xen/xen_apic.c +++ b/sys/x86/xen/xen_apic.c @@ -107,6 +107,14 @@ static struct xen_ipi_handler xen_ipis[] = }; #endif +/* + * Save previous (native) handler as a fallback. Xen < 4.7 doesn't support + * VCPUOP_send_nmi for HVM guests, and thus we need a fallback in that case: + * + * https://lists.freebsd.org/archives/freebsd-xen/2022-January/000032.html + */ +void (*native_ipi_vectored)(u_int, int); + /*------------------------------- Per-CPU Data -------------------------------*/ #ifdef SMP DPCPU_DEFINE(xen_intr_handle_t, ipi_handle[nitems(xen_ipis)]); @@ -273,10 +281,11 @@ xen_pv_lapic_ipi_raw(register_t icrlo, u_int dest) } #define PCPU_ID_GET(id, field) (pcpu_find(id)->pc_##field) -static void +static int send_nmi(int dest) { unsigned int cpu; + int rc = 0; /* * NMIs are not routed over event channels, and instead delivered as on @@ -286,24 +295,33 @@ send_nmi(int dest) */ switch(dest) { case APIC_IPI_DEST_SELF: - HYPERVISOR_vcpu_op(VCPUOP_send_nmi, PCPU_GET(vcpu_id), NULL); + rc = HYPERVISOR_vcpu_op(VCPUOP_send_nmi, PCPU_GET(vcpu_id), NULL); break; case APIC_IPI_DEST_ALL: - CPU_FOREACH(cpu) - HYPERVISOR_vcpu_op(VCPUOP_send_nmi, + CPU_FOREACH(cpu) { + rc = HYPERVISOR_vcpu_op(VCPUOP_send_nmi, PCPU_ID_GET(cpu, vcpu_id), NULL); + if (rc != 0) + break; + } break; case APIC_IPI_DEST_OTHERS: - CPU_FOREACH(cpu) - if (cpu != PCPU_GET(cpuid)) - HYPERVISOR_vcpu_op(VCPUOP_send_nmi, + CPU_FOREACH(cpu) { + if (cpu != PCPU_GET(cpuid)) { + rc = HYPERVISOR_vcpu_op(VCPUOP_send_nmi, PCPU_ID_GET(cpu, vcpu_id), NULL); + if (rc != 0) + break; + } + } break; default: - HYPERVISOR_vcpu_op(VCPUOP_send_nmi, + rc = HYPERVISOR_vcpu_op(VCPUOP_send_nmi, PCPU_ID_GET(apic_cpuid(dest), vcpu_id), NULL); break; } + + return rc; } #undef PCPU_ID_GET @@ -312,9 +330,21 @@ xen_pv_lapic_ipi_vectored(u_int vector, int dest) { xen_intr_handle_t *ipi_handle; int ipi_idx, to_cpu, self; + static bool pvnmi = true; if (vector >= IPI_NMI_FIRST) { - send_nmi(dest); + if (pvnmi) { + int rc = send_nmi(dest); + + if (rc != 0) { + printf( + "Sending NMI using hypercall failed (%d) switching to APIC\n", rc); + pvnmi = false; + native_ipi_vectored(vector, dest); + } + } else + native_ipi_vectored(vector, dest); + return; } @@ -580,8 +610,8 @@ xen_setup_cpus(void) xen_cpu_ipi_init(i); /* Set the xen pv ipi ops to replace the native ones */ - if (xen_hvm_domain()) - apic_ops.ipi_vectored = xen_pv_lapic_ipi_vectored; + native_ipi_vectored = apic_ops.ipi_vectored; + apic_ops.ipi_vectored = xen_pv_lapic_ipi_vectored; } /* Switch to using PV IPIs as soon as the vcpu_id is set. */