From nobody Fri Jan 28 10:34:34 2022 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 0D6A3197D631; Fri, 28 Jan 2022 10:34:35 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JlYjk5LZ2z3tLY; Fri, 28 Jan 2022 10:34:34 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1643366074; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=8Zgk/psk/wEfXH6zciPBVVwTOaoPwTJSDhTIQG7Z/JY=; b=vwLvc27D9pIn42xyg7h598IyCOis92eX3PvQqAKS5WSMShYu98UsGosb+35+Ovgrkqxb1N meVFDlfYVkLi8Ds6NxgLQLTnQw+c36HshbNrq9k0benHmwMrok/3SWkKZ5JEm+CT1+/XyX +s7hmC3mn27iULQJjWue1DCSl7JlDo5ka+RGD8+ONJlI4+tPh5gGKcURIUnsuucO4f6Tna poFtu4IRy/GbOedxUUTqOYVZFPcL476l+R/vCEdq8GonFbsmaw7HafLyVfhvXtxSTcvzGb ajnvmP9ywpHDQD9eUuBpkXHSdSwoJWHYMMR5Cag1Ow9haDxvcWlUd9FhEO8RPw== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 928FB287BC; Fri, 28 Jan 2022 10:34:34 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 20SAYYHU058755; Fri, 28 Jan 2022 10:34:34 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 20SAYYSO058754; Fri, 28 Jan 2022 10:34:34 GMT (envelope-from git) Date: Fri, 28 Jan 2022 10:34:34 GMT Message-Id: <202201281034.20SAYYSO058754@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Emmanuel Vadot Subject: git: 24100ad5de39 - stable/13 - bhyve: add support for MTRR List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-branches@freebsd.org X-BeenThere: dev-commits-src-branches@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: manu X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: 24100ad5de3933fc95726f33d9b6df964b779f53 Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1643366075; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=8Zgk/psk/wEfXH6zciPBVVwTOaoPwTJSDhTIQG7Z/JY=; b=xC5MOf+5Wpv5KRIVXlZMH/xWrZfQXB80hYk0Xs1mSw6b96ae91P+oR8aHJAAsden4hiOY9 yXPKANeXmYjNIKmvY7C8akeZXEIGERYEIDaI9GcM34cSR3zEEduxExSMQwMvmKZnZyR9zF 7eo8PY/5AY8aVi3f1C4R2x3Lifnn2smMojL2QvTDpU06iSvG33mwWhP3RHY5THIeTNZG2Q AfPzL+TzqdRpK/m46a23osY6WuWKL1ZbeUwi/0xHXsKsxd9emUNARq532oHUyl7oLv0ZIz fdf86u08YUsYLUHyuVjFaFk62cHCE++FVNZGZYBRcmXUkGOH681V4zbDYR+Vww== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1643366075; a=rsa-sha256; cv=none; b=d0ESCD2pVXSIZPb1wuLtRt++fKQuSckeOWpUicne3mVniaVzCJ97mQvoyqNY5ve3d8Klm7 3Ldy2WtpVwR6ipV0GyHu9lake6beKaeb0ylPqAbHHwqGyuTXEmrTBjNRTT9qqUk2CCvbw5 G6v6u/ZHcQBKWLPjNOjxiF7LSrKI8lbTMKQX2oxCTdu4ZcHDw+QSN5F5yzLn+UdAzpuD9b 9jNz4bhSAX8J3nvMgw3dHxCgMp7x0QcIZ41I3LP9Ylz6ZUS6QeRCxiSJqYT1mh9s3nTGPJ SiQKnSdzOyr6dJSxv7zJdEvwwvVJJifrn858y9wAmZsoYIDuym4ztGlFFJTgPQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch stable/13 has been updated by manu: URL: https://cgit.FreeBSD.org/src/commit/?id=24100ad5de3933fc95726f33d9b6df964b779f53 commit 24100ad5de3933fc95726f33d9b6df964b779f53 Author: Corvin Köhne AuthorDate: 2022-01-14 09:58:48 +0000 Commit: Emmanuel Vadot CommitDate: 2022-01-28 07:10:56 +0000 bhyve: add support for MTRR Some guests or driver might depend on MTRR to work properly. E.g. the nvidia gpu driver won't work without MTRR. Reviewed by: markj MFC after: 2 weeks Sponsored by: Beckhoff Automation GmbH & Co. KG Differential Revision: https://reviews.freebsd.org/D33333 (cherry picked from commit 6171e026be11824495cebe8baf559af673a8e533) --- sys/amd64/vmm/amd/svm_msr.c | 16 ++++++--- sys/amd64/vmm/amd/svm_softc.h | 3 ++ sys/amd64/vmm/intel/vmx.h | 2 ++ sys/amd64/vmm/intel/vmx_msr.c | 17 +++++---- sys/amd64/vmm/x86.c | 82 +++++++++++++++++++++++++++++++++++++++++++ sys/amd64/vmm/x86.h | 20 +++++++++++ 6 files changed, 130 insertions(+), 10 deletions(-) diff --git a/sys/amd64/vmm/amd/svm_msr.c b/sys/amd64/vmm/amd/svm_msr.c index 12046de4dbb9..1a22f16cf48e 100644 --- a/sys/amd64/vmm/amd/svm_msr.c +++ b/sys/amd64/vmm/amd/svm_msr.c @@ -120,9 +120,14 @@ svm_rdmsr(struct svm_softc *sc, int vcpu, u_int num, uint64_t *result, break; case MSR_MTRRcap: case MSR_MTRRdefType: - case MSR_MTRR4kBase ... MSR_MTRR4kBase + 8: + case MSR_MTRR4kBase ... MSR_MTRR4kBase + 7: case MSR_MTRR16kBase ... MSR_MTRR16kBase + 1: case MSR_MTRR64kBase: + case MSR_MTRRVarBase ... MSR_MTRRVarBase + (VMM_MTRR_VAR_MAX * 2) - 1: + if (vm_rdmtrr(&sc->mtrr[vcpu], num, result) != 0) { + vm_inject_gp(sc->vm, vcpu); + } + break; case MSR_SYSCFG: case MSR_AMDK8_IPM: case MSR_EXTFEATURES: @@ -146,12 +151,15 @@ svm_wrmsr(struct svm_softc *sc, int vcpu, u_int num, uint64_t val, bool *retu) case MSR_MCG_STATUS: break; /* ignore writes */ case MSR_MTRRcap: - vm_inject_gp(sc->vm, vcpu); - break; case MSR_MTRRdefType: - case MSR_MTRR4kBase ... MSR_MTRR4kBase + 8: + case MSR_MTRR4kBase ... MSR_MTRR4kBase + 7: case MSR_MTRR16kBase ... MSR_MTRR16kBase + 1: case MSR_MTRR64kBase: + case MSR_MTRRVarBase ... MSR_MTRRVarBase + (VMM_MTRR_VAR_MAX * 2) - 1: + if (vm_wrmtrr(&sc->mtrr[vcpu], num, val) != 0) { + vm_inject_gp(sc->vm, vcpu); + } + break; case MSR_SYSCFG: break; /* Ignore writes */ case MSR_AMDK8_IPM: diff --git a/sys/amd64/vmm/amd/svm_softc.h b/sys/amd64/vmm/amd/svm_softc.h index 8735353bb472..5f6a267617d2 100644 --- a/sys/amd64/vmm/amd/svm_softc.h +++ b/sys/amd64/vmm/amd/svm_softc.h @@ -31,6 +31,8 @@ #ifndef _SVM_SOFTC_H_ #define _SVM_SOFTC_H_ +#include "x86.h" + #define SVM_IO_BITMAP_SIZE (3 * PAGE_SIZE) #define SVM_MSR_BITMAP_SIZE (2 * PAGE_SIZE) @@ -64,6 +66,7 @@ struct svm_softc { uint8_t *iopm_bitmap; /* shared by all vcpus */ uint8_t *msr_bitmap; /* shared by all vcpus */ struct vm *vm; + struct vm_mtrr mtrr[VM_MAXCPU]; }; CTASSERT((offsetof(struct svm_softc, nptp) & PAGE_MASK) == 0); diff --git a/sys/amd64/vmm/intel/vmx.h b/sys/amd64/vmm/intel/vmx.h index 57499a3a6869..81e508e30d3d 100644 --- a/sys/amd64/vmm/intel/vmx.h +++ b/sys/amd64/vmm/intel/vmx.h @@ -32,6 +32,7 @@ #define _VMX_H_ #include "vmcs.h" +#include "x86.h" struct pmap; @@ -134,6 +135,7 @@ struct vmx { uint64_t eptp; struct vm *vm; long eptgen[MAXCPU]; /* cached pmap->pm_eptgen */ + struct vm_mtrr mtrr[VM_MAXCPU]; }; CTASSERT((offsetof(struct vmx, vmcs) & PAGE_MASK) == 0); CTASSERT((offsetof(struct vmx, msr_bitmap) & PAGE_MASK) == 0); diff --git a/sys/amd64/vmm/intel/vmx_msr.c b/sys/amd64/vmm/intel/vmx_msr.c index db9e659bab5b..a135518cb1c3 100644 --- a/sys/amd64/vmm/intel/vmx_msr.c +++ b/sys/amd64/vmm/intel/vmx_msr.c @@ -425,10 +425,13 @@ vmx_rdmsr(struct vmx *vmx, int vcpuid, u_int num, uint64_t *val, bool *retu) break; case MSR_MTRRcap: case MSR_MTRRdefType: - case MSR_MTRR4kBase ... MSR_MTRR4kBase + 8: + case MSR_MTRR4kBase ... MSR_MTRR4kBase + 7: case MSR_MTRR16kBase ... MSR_MTRR16kBase + 1: case MSR_MTRR64kBase: - *val = 0; + case MSR_MTRRVarBase ... MSR_MTRRVarBase + (VMM_MTRR_VAR_MAX * 2) - 1: + if (vm_rdmtrr(&vmx->mtrr[vcpuid], num, val) != 0) { + vm_inject_gp(vmx->vm, vcpuid); + } break; case MSR_IA32_MISC_ENABLE: *val = misc_enable; @@ -465,13 +468,15 @@ vmx_wrmsr(struct vmx *vmx, int vcpuid, u_int num, uint64_t val, bool *retu) case MSR_MCG_STATUS: break; /* ignore writes */ case MSR_MTRRcap: - vm_inject_gp(vmx->vm, vcpuid); - break; case MSR_MTRRdefType: - case MSR_MTRR4kBase ... MSR_MTRR4kBase + 8: + case MSR_MTRR4kBase ... MSR_MTRR4kBase + 7: case MSR_MTRR16kBase ... MSR_MTRR16kBase + 1: case MSR_MTRR64kBase: - break; /* Ignore writes */ + case MSR_MTRRVarBase ... MSR_MTRRVarBase + (VMM_MTRR_VAR_MAX * 2) - 1: + if (vm_wrmtrr(&vmx->mtrr[vcpuid], num, val) != 0) { + vm_inject_gp(vmx->vm, vcpuid); + } + break; case MSR_IA32_MISC_ENABLE: changed = val ^ misc_enable; /* diff --git a/sys/amd64/vmm/x86.c b/sys/amd64/vmm/x86.c index c43a3c870211..c97cb91af4f6 100644 --- a/sys/amd64/vmm/x86.c +++ b/sys/amd64/vmm/x86.c @@ -648,3 +648,85 @@ vm_cpuid_capability(struct vm *vm, int vcpuid, enum vm_cpuid_capability cap) } return (rv); } + +int +vm_rdmtrr(struct vm_mtrr *mtrr, u_int num, uint64_t *val) +{ + switch (num) { + case MSR_MTRRcap: + *val = MTRR_CAP_WC | MTRR_CAP_FIXED | VMM_MTRR_VAR_MAX; + break; + case MSR_MTRRdefType: + *val = mtrr->def_type; + break; + case MSR_MTRR4kBase ... MSR_MTRR4kBase + 7: + *val = mtrr->fixed4k[num - MSR_MTRR4kBase]; + break; + case MSR_MTRR16kBase ... MSR_MTRR16kBase + 1: + *val = mtrr->fixed16k[num - MSR_MTRR16kBase]; + break; + case MSR_MTRR64kBase: + *val = mtrr->fixed64k; + break; + case MSR_MTRRVarBase ... MSR_MTRRVarBase + (VMM_MTRR_VAR_MAX * 2) - 1: { + u_int offset = num - MSR_MTRRVarBase; + if (offset % 2 == 0) { + *val = mtrr->var[offset / 2].base; + } else { + *val = mtrr->var[offset / 2].mask; + } + break; + } + default: + return (-1); + } + + return (0); +} + +int +vm_wrmtrr(struct vm_mtrr *mtrr, u_int num, uint64_t val) +{ + switch (num) { + case MSR_MTRRcap: + /* MTRRCAP is read only */ + return (-1); + case MSR_MTRRdefType: + if (val & ~VMM_MTRR_DEF_MASK) { + /* generate #GP on writes to reserved fields */ + return (-1); + } + mtrr->def_type = val; + break; + case MSR_MTRR4kBase ... MSR_MTRR4kBase + 7: + mtrr->fixed4k[num - MSR_MTRR4kBase] = val; + break; + case MSR_MTRR16kBase ... MSR_MTRR16kBase + 1: + mtrr->fixed16k[num - MSR_MTRR16kBase] = val; + break; + case MSR_MTRR64kBase: + mtrr->fixed64k = val; + break; + case MSR_MTRRVarBase ... MSR_MTRRVarBase + (VMM_MTRR_VAR_MAX * 2) - 1: { + u_int offset = num - MSR_MTRRVarBase; + if (offset % 2 == 0) { + if (val & ~VMM_MTRR_PHYSBASE_MASK) { + /* generate #GP on writes to reserved fields */ + return (-1); + } + mtrr->var[offset / 2].base = val; + } else { + if (val & ~VMM_MTRR_PHYSMASK_MASK) { + /* generate #GP on writes to reserved fields */ + return (-1); + } + mtrr->var[offset / 2].mask = val; + } + break; + } + default: + return (-1); + } + + return (0); +} diff --git a/sys/amd64/vmm/x86.h b/sys/amd64/vmm/x86.h index 7c8fccf78f28..318f0b5cf871 100644 --- a/sys/amd64/vmm/x86.h +++ b/sys/amd64/vmm/x86.h @@ -80,4 +80,24 @@ enum vm_cpuid_capability { * and 'false' otherwise. */ bool vm_cpuid_capability(struct vm *vm, int vcpuid, enum vm_cpuid_capability); + +#define VMM_MTRR_VAR_MAX 10 +#define VMM_MTRR_DEF_MASK \ + (MTRR_DEF_ENABLE | MTRR_DEF_FIXED_ENABLE | MTRR_DEF_TYPE) +#define VMM_MTRR_PHYSBASE_MASK (MTRR_PHYSBASE_PHYSBASE | MTRR_PHYSBASE_TYPE) +#define VMM_MTRR_PHYSMASK_MASK (MTRR_PHYSMASK_PHYSMASK | MTRR_PHYSMASK_VALID) +struct vm_mtrr { + uint64_t def_type; + uint64_t fixed4k[8]; + uint64_t fixed16k[2]; + uint64_t fixed64k; + struct { + uint64_t base; + uint64_t mask; + } var[VMM_MTRR_VAR_MAX]; +}; + +int vm_rdmtrr(struct vm_mtrr *mtrr, u_int num, uint64_t *val); +int vm_wrmtrr(struct vm_mtrr *mtrr, u_int num, uint64_t val); + #endif