From nobody Fri Jan 14 11:42:11 2022 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 823F3195FC1C; Fri, 14 Jan 2022 11:42:11 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JZztC3GrKz3MMm; Fri, 14 Jan 2022 11:42:11 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1642160531; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=jEHuthooV8JYt0zy1f7FIJgRDBnRnodCWtbWmwZZdOk=; b=oMAFNYLEHCR/BujXDlKEuKNz4FPCWXyqe3Cjh1TsnvOGZE3SLep0WrJUYvysWFfijhun38 SxrC1dRU2myBeV3FyiHhcWxdzM/0y3ci1EyEj6qEgALYbcH9NCOio+Dvqw3rdPEUDvcX6r br4/aoL1+CqHh96Ngberk20qP5ossvHF+Q+7o9avxAelF20buELg9I1EuIJC27h5jNku0F UauWRRMlLtH2BtY10iKcvPH7AUpEFf6unnD1/gGxSgT6HU8ZfOdYGyHm4CGQ12XbhF5kM1 gqTfKLR1Eq3edZeWWSL14nbFpaX3kiQaQQisBg3WNRNtbfgZgcZYtn3BTCPK3w== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 503641346E; Fri, 14 Jan 2022 11:42:11 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 20EBgBQV076922; Fri, 14 Jan 2022 11:42:11 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 20EBgBdI076921; Fri, 14 Jan 2022 11:42:11 GMT (envelope-from git) Date: Fri, 14 Jan 2022 11:42:11 GMT Message-Id: <202201141142.20EBgBdI076921@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Emmanuel Vadot Subject: git: 6171e026be11 - main - bhyve: add support for MTRR List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: manu X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 6171e026be11824495cebe8baf559af673a8e533 Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1642160531; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=jEHuthooV8JYt0zy1f7FIJgRDBnRnodCWtbWmwZZdOk=; b=bmUPYsqkyxIptplSyI3+dIcl9vRduRCoiggvY2203hU8LbT1m0B/SSZkkrP7A9GhGS1dsj 2bhGOiCwtQRcmLUPMaKQq0cMSW3FpUkVdQDE1lVsxAeE0nbfTjWT3izzKQQYYlvLgB7sV/ ubTLt2jUGEIqpIzz1tjWvtiBw/9paMLiIXx6PG8W1uY/J2czztC1/ZG12UsdS1Z0IdmLDm mtJCxFmIIW/R8vp+AZcLV3kJhR5iDFrcQ67qToU/1hJosI8avGKpq8noN/MMNoLrLZE3tS bWdw8t++SBPWLfj3TxO/hX1CbcjorhOcnseLw+htqGsjgDBOjnOb1Kxk9ET0mw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1642160531; a=rsa-sha256; cv=none; b=mH+4HYPQtupYPVWYyYS0o04tCpEo8mIPcMD7gDp8Q1GtL6E4U/TyWqNheI3G4nGJbgr53a moPFgv+G+Za4Pw/2u+ZDwpSzc7KjuVZV9vyVLDWn00vHqSly7heixsh7k7+75Mv8Ly/Ai0 VBPbNu8H2v0HtJmwQW6mXf+moHt0F0Cfc8QtXnsScDQWaWRVzO/lCwwOP5+ZuLnG794rQ7 V/vBqTE35ZRlfFmoh3fE9Hfl5w49boiMt0ZU2HGhHLKisQlcWX6bdqGdOmL32QNlKFSmJp 6P2maMKazYkbTG8uv3bEWQAN+VDSb9RPyjmLYLQ0FwZ0Zx00ycnVMJEVui9yQQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by manu: URL: https://cgit.FreeBSD.org/src/commit/?id=6171e026be11824495cebe8baf559af673a8e533 commit 6171e026be11824495cebe8baf559af673a8e533 Author: Corvin Köhne AuthorDate: 2022-01-14 09:58:48 +0000 Commit: Emmanuel Vadot CommitDate: 2022-01-14 11:41:44 +0000 bhyve: add support for MTRR Some guests or driver might depend on MTRR to work properly. E.g. the nvidia gpu driver won't work without MTRR. Reviewed by: markj MFC after: 2 weeks Sponsored by: Beckhoff Automation GmbH & Co. KG Differential Revision: https://reviews.freebsd.org/D33333 --- sys/amd64/vmm/amd/svm_msr.c | 16 ++++++--- sys/amd64/vmm/amd/svm_softc.h | 3 ++ sys/amd64/vmm/intel/vmx.h | 2 ++ sys/amd64/vmm/intel/vmx_msr.c | 17 +++++---- sys/amd64/vmm/x86.c | 82 +++++++++++++++++++++++++++++++++++++++++++ sys/amd64/vmm/x86.h | 20 +++++++++++ 6 files changed, 130 insertions(+), 10 deletions(-) diff --git a/sys/amd64/vmm/amd/svm_msr.c b/sys/amd64/vmm/amd/svm_msr.c index 12046de4dbb9..1a22f16cf48e 100644 --- a/sys/amd64/vmm/amd/svm_msr.c +++ b/sys/amd64/vmm/amd/svm_msr.c @@ -120,9 +120,14 @@ svm_rdmsr(struct svm_softc *sc, int vcpu, u_int num, uint64_t *result, break; case MSR_MTRRcap: case MSR_MTRRdefType: - case MSR_MTRR4kBase ... MSR_MTRR4kBase + 8: + case MSR_MTRR4kBase ... MSR_MTRR4kBase + 7: case MSR_MTRR16kBase ... MSR_MTRR16kBase + 1: case MSR_MTRR64kBase: + case MSR_MTRRVarBase ... MSR_MTRRVarBase + (VMM_MTRR_VAR_MAX * 2) - 1: + if (vm_rdmtrr(&sc->mtrr[vcpu], num, result) != 0) { + vm_inject_gp(sc->vm, vcpu); + } + break; case MSR_SYSCFG: case MSR_AMDK8_IPM: case MSR_EXTFEATURES: @@ -146,12 +151,15 @@ svm_wrmsr(struct svm_softc *sc, int vcpu, u_int num, uint64_t val, bool *retu) case MSR_MCG_STATUS: break; /* ignore writes */ case MSR_MTRRcap: - vm_inject_gp(sc->vm, vcpu); - break; case MSR_MTRRdefType: - case MSR_MTRR4kBase ... MSR_MTRR4kBase + 8: + case MSR_MTRR4kBase ... MSR_MTRR4kBase + 7: case MSR_MTRR16kBase ... MSR_MTRR16kBase + 1: case MSR_MTRR64kBase: + case MSR_MTRRVarBase ... MSR_MTRRVarBase + (VMM_MTRR_VAR_MAX * 2) - 1: + if (vm_wrmtrr(&sc->mtrr[vcpu], num, val) != 0) { + vm_inject_gp(sc->vm, vcpu); + } + break; case MSR_SYSCFG: break; /* Ignore writes */ case MSR_AMDK8_IPM: diff --git a/sys/amd64/vmm/amd/svm_softc.h b/sys/amd64/vmm/amd/svm_softc.h index 8735353bb472..5f6a267617d2 100644 --- a/sys/amd64/vmm/amd/svm_softc.h +++ b/sys/amd64/vmm/amd/svm_softc.h @@ -31,6 +31,8 @@ #ifndef _SVM_SOFTC_H_ #define _SVM_SOFTC_H_ +#include "x86.h" + #define SVM_IO_BITMAP_SIZE (3 * PAGE_SIZE) #define SVM_MSR_BITMAP_SIZE (2 * PAGE_SIZE) @@ -64,6 +66,7 @@ struct svm_softc { uint8_t *iopm_bitmap; /* shared by all vcpus */ uint8_t *msr_bitmap; /* shared by all vcpus */ struct vm *vm; + struct vm_mtrr mtrr[VM_MAXCPU]; }; CTASSERT((offsetof(struct svm_softc, nptp) & PAGE_MASK) == 0); diff --git a/sys/amd64/vmm/intel/vmx.h b/sys/amd64/vmm/intel/vmx.h index 57499a3a6869..81e508e30d3d 100644 --- a/sys/amd64/vmm/intel/vmx.h +++ b/sys/amd64/vmm/intel/vmx.h @@ -32,6 +32,7 @@ #define _VMX_H_ #include "vmcs.h" +#include "x86.h" struct pmap; @@ -134,6 +135,7 @@ struct vmx { uint64_t eptp; struct vm *vm; long eptgen[MAXCPU]; /* cached pmap->pm_eptgen */ + struct vm_mtrr mtrr[VM_MAXCPU]; }; CTASSERT((offsetof(struct vmx, vmcs) & PAGE_MASK) == 0); CTASSERT((offsetof(struct vmx, msr_bitmap) & PAGE_MASK) == 0); diff --git a/sys/amd64/vmm/intel/vmx_msr.c b/sys/amd64/vmm/intel/vmx_msr.c index db9e659bab5b..a135518cb1c3 100644 --- a/sys/amd64/vmm/intel/vmx_msr.c +++ b/sys/amd64/vmm/intel/vmx_msr.c @@ -425,10 +425,13 @@ vmx_rdmsr(struct vmx *vmx, int vcpuid, u_int num, uint64_t *val, bool *retu) break; case MSR_MTRRcap: case MSR_MTRRdefType: - case MSR_MTRR4kBase ... MSR_MTRR4kBase + 8: + case MSR_MTRR4kBase ... MSR_MTRR4kBase + 7: case MSR_MTRR16kBase ... MSR_MTRR16kBase + 1: case MSR_MTRR64kBase: - *val = 0; + case MSR_MTRRVarBase ... MSR_MTRRVarBase + (VMM_MTRR_VAR_MAX * 2) - 1: + if (vm_rdmtrr(&vmx->mtrr[vcpuid], num, val) != 0) { + vm_inject_gp(vmx->vm, vcpuid); + } break; case MSR_IA32_MISC_ENABLE: *val = misc_enable; @@ -465,13 +468,15 @@ vmx_wrmsr(struct vmx *vmx, int vcpuid, u_int num, uint64_t val, bool *retu) case MSR_MCG_STATUS: break; /* ignore writes */ case MSR_MTRRcap: - vm_inject_gp(vmx->vm, vcpuid); - break; case MSR_MTRRdefType: - case MSR_MTRR4kBase ... MSR_MTRR4kBase + 8: + case MSR_MTRR4kBase ... MSR_MTRR4kBase + 7: case MSR_MTRR16kBase ... MSR_MTRR16kBase + 1: case MSR_MTRR64kBase: - break; /* Ignore writes */ + case MSR_MTRRVarBase ... MSR_MTRRVarBase + (VMM_MTRR_VAR_MAX * 2) - 1: + if (vm_wrmtrr(&vmx->mtrr[vcpuid], num, val) != 0) { + vm_inject_gp(vmx->vm, vcpuid); + } + break; case MSR_IA32_MISC_ENABLE: changed = val ^ misc_enable; /* diff --git a/sys/amd64/vmm/x86.c b/sys/amd64/vmm/x86.c index c43a3c870211..c97cb91af4f6 100644 --- a/sys/amd64/vmm/x86.c +++ b/sys/amd64/vmm/x86.c @@ -648,3 +648,85 @@ vm_cpuid_capability(struct vm *vm, int vcpuid, enum vm_cpuid_capability cap) } return (rv); } + +int +vm_rdmtrr(struct vm_mtrr *mtrr, u_int num, uint64_t *val) +{ + switch (num) { + case MSR_MTRRcap: + *val = MTRR_CAP_WC | MTRR_CAP_FIXED | VMM_MTRR_VAR_MAX; + break; + case MSR_MTRRdefType: + *val = mtrr->def_type; + break; + case MSR_MTRR4kBase ... MSR_MTRR4kBase + 7: + *val = mtrr->fixed4k[num - MSR_MTRR4kBase]; + break; + case MSR_MTRR16kBase ... MSR_MTRR16kBase + 1: + *val = mtrr->fixed16k[num - MSR_MTRR16kBase]; + break; + case MSR_MTRR64kBase: + *val = mtrr->fixed64k; + break; + case MSR_MTRRVarBase ... MSR_MTRRVarBase + (VMM_MTRR_VAR_MAX * 2) - 1: { + u_int offset = num - MSR_MTRRVarBase; + if (offset % 2 == 0) { + *val = mtrr->var[offset / 2].base; + } else { + *val = mtrr->var[offset / 2].mask; + } + break; + } + default: + return (-1); + } + + return (0); +} + +int +vm_wrmtrr(struct vm_mtrr *mtrr, u_int num, uint64_t val) +{ + switch (num) { + case MSR_MTRRcap: + /* MTRRCAP is read only */ + return (-1); + case MSR_MTRRdefType: + if (val & ~VMM_MTRR_DEF_MASK) { + /* generate #GP on writes to reserved fields */ + return (-1); + } + mtrr->def_type = val; + break; + case MSR_MTRR4kBase ... MSR_MTRR4kBase + 7: + mtrr->fixed4k[num - MSR_MTRR4kBase] = val; + break; + case MSR_MTRR16kBase ... MSR_MTRR16kBase + 1: + mtrr->fixed16k[num - MSR_MTRR16kBase] = val; + break; + case MSR_MTRR64kBase: + mtrr->fixed64k = val; + break; + case MSR_MTRRVarBase ... MSR_MTRRVarBase + (VMM_MTRR_VAR_MAX * 2) - 1: { + u_int offset = num - MSR_MTRRVarBase; + if (offset % 2 == 0) { + if (val & ~VMM_MTRR_PHYSBASE_MASK) { + /* generate #GP on writes to reserved fields */ + return (-1); + } + mtrr->var[offset / 2].base = val; + } else { + if (val & ~VMM_MTRR_PHYSMASK_MASK) { + /* generate #GP on writes to reserved fields */ + return (-1); + } + mtrr->var[offset / 2].mask = val; + } + break; + } + default: + return (-1); + } + + return (0); +} diff --git a/sys/amd64/vmm/x86.h b/sys/amd64/vmm/x86.h index 7c8fccf78f28..318f0b5cf871 100644 --- a/sys/amd64/vmm/x86.h +++ b/sys/amd64/vmm/x86.h @@ -80,4 +80,24 @@ enum vm_cpuid_capability { * and 'false' otherwise. */ bool vm_cpuid_capability(struct vm *vm, int vcpuid, enum vm_cpuid_capability); + +#define VMM_MTRR_VAR_MAX 10 +#define VMM_MTRR_DEF_MASK \ + (MTRR_DEF_ENABLE | MTRR_DEF_FIXED_ENABLE | MTRR_DEF_TYPE) +#define VMM_MTRR_PHYSBASE_MASK (MTRR_PHYSBASE_PHYSBASE | MTRR_PHYSBASE_TYPE) +#define VMM_MTRR_PHYSMASK_MASK (MTRR_PHYSMASK_PHYSMASK | MTRR_PHYSMASK_VALID) +struct vm_mtrr { + uint64_t def_type; + uint64_t fixed4k[8]; + uint64_t fixed16k[2]; + uint64_t fixed64k; + struct { + uint64_t base; + uint64_t mask; + } var[VMM_MTRR_VAR_MAX]; +}; + +int vm_rdmtrr(struct vm_mtrr *mtrr, u_int num, uint64_t *val); +int vm_wrmtrr(struct vm_mtrr *mtrr, u_int num, uint64_t val); + #endif