Re: git: 6cf00ef80c20 - main - arm64: enablement for ARM64 in Hyper-V (Part 1)
- In reply to: Andrew Turner : "Re: git: 6cf00ef80c20 - main - arm64: enablement for ARM64 in Hyper-V (Part 1)"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Thu, 29 Sep 2022 14:19:44 UTC
On 29 Sept 2022, at 15:12, Andrew Turner <andrew@fubar.geek.nz> wrote:
>
> It looks like you have the contents of sys/dev/hyperv/vmbus/aarch64/hyperv_aarch64.c duplicated.
3 of the 5 files in fact:
https://reviews.freebsd.org/D36466?vs=111054&id=111175#toc
Also confused why both aarch64 and machdep files exist; surely that’s
the same concept? Also strange to land aarch64 files without hooking
them up in sys/conf and sys/modules.
Jess
>> On 29 Sep 2022, at 11:19, Wei Hu <whu@FreeBSD.org> wrote:
>>
>> The branch main has been updated by whu:
>>
>> URL: https://cgit.FreeBSD.org/src/commit/?id=6cf00ef80c2014811bb6ce35e969b95dfe574793
>>
>> commit 6cf00ef80c2014811bb6ce35e969b95dfe574793
>> Author: Souradeep Chakrabarti <schakrabarti@microsoft.com>
>> AuthorDate: 2022-09-29 09:57:46 +0000
>> Commit: Wei Hu <whu@FreeBSD.org>
>> CommitDate: 2022-09-29 10:14:41 +0000
>>
>> arm64: enablement for ARM64 in Hyper-V (Part 1)
>>
>> The changes are to refactor the code of vmbus.c and hyperv.c to keep minimal
>> arch specific codes there and have them in separate files in x86/ arm64/ .
>> x86 is a new directory, which contains codes for x86 / x86_64. Instead of
>> repeating the same codes in existing amd64/ and i386/, this approach reduced
>> the repetition. This is first of three patches for Hyper-V enablement.
>>
>> Reviewed by: whu
>> Tested by: Souradeep Chakrabarti <schakrabarti@microsoft.com>
>> Sponsored by: Microsoft
>> Differential Revision: https://reviews.freebsd.org/D36466
>> ---
>> sys/dev/hyperv/vmbus/aarch64/hyperv_aarch64.c | 190 +++++++++++++
>> sys/dev/hyperv/vmbus/aarch64/hyperv_machdep.c | 103 +++++++
>> sys/dev/hyperv/vmbus/aarch64/hyperv_machdep.h | 59 ++++
>> sys/dev/hyperv/vmbus/aarch64/hyperv_reg.h | 386 ++++++++++++++++++++++++++
>> sys/dev/hyperv/vmbus/aarch64/vmbus_aarch64.c | 314 +++++++++++++++++++++
>> 5 files changed, 1052 insertions(+)
>>
>> diff --git a/sys/dev/hyperv/vmbus/aarch64/hyperv_aarch64.c b/sys/dev/hyperv/vmbus/aarch64/hyperv_aarch64.c
>> new file mode 100644
>> index 000000000000..6104e88130db
>> --- /dev/null
>> +++ b/sys/dev/hyperv/vmbus/aarch64/hyperv_aarch64.c
>> @@ -0,0 +1,190 @@
>> +/*- SPDX-License-Identifier: BSD-2-Clause-FreeBSD
>> + * Copyright (c) 2009-2012,2016-2017, 2022 Microsoft Corp.
>> + * Copyright (c) 2012 NetApp Inc.
>> + * Copyright (c) 2012 Citrix Inc.
>> + * All rights reserved.
>> + *
>> + * Redistribution and use in source and binary forms, with or without
>> + * modification, are permitted provided that the following conditions
>> + * are met:
>> + * 1. Redistributions of source code must retain the above copyright
>> + * notice unmodified, this list of conditions, and the following
>> + * disclaimer.
>> + * 2. Redistributions in binary form must reproduce the above copyright
>> + * notice, this list of conditions and the following disclaimer in the
>> + * documentation and/or other materials provided with the distribution.
>> + *
>> + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
>> + * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
>> + * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
>> + * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
>> + * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
>> + * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
>> + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
>> + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>> + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
>> + * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
>> + */
>> +
>> +/**
>> + * Implements low-level interactions with Hyper-V/Azure
>> + */
>> +
>> +#include <sys/cdefs.h>
>> +__FBSDID("$FreeBSD$");
>> +
>> +#include <sys/param.h>
>> +#include <sys/systm.h>
>> +#include <sys/kernel.h>
>> +#include <sys/malloc.h>
>> +#include <sys/timetc.h>
>> +
>> +#include <vm/vm.h>
>> +#include <vm/pmap.h>
>> +#include <vm/vm_extern.h>
>> +#include <vm/vm_kern.h>
>> +
>> +#include <dev/hyperv/include/hyperv.h>
>> +#include <dev/hyperv/include/hyperv_busdma.h>
>> +#include <dev/hyperv/vmbus/aarch64/hyperv_machdep.h>
>> +#include <dev/hyperv/vmbus/aarch64/hyperv_reg.h>
>> +#include <dev/hyperv/vmbus/hyperv_var.h>
>> +
>> +void hyperv_init_tc(void);
>> +int hypercall_page_setup(vm_paddr_t);
>> +void hypercall_disable(void);
>> +bool hyperv_identify_features(void);
>> +
>> +u_int hyperv_ver_major;
>> +u_int hyperv_features;
>> +u_int hyperv_recommends;
>> +
>> +hyperv_tc64_t hyperv_tc64;
>> +
>> +void
>> +hyperv_init_tc(void)
>> +{
>> + hyperv_tc64 = NULL;
>> +}
>> +
>> +int
>> +hypercall_page_setup(vm_paddr_t hc)
>> +{
>> + return (0);
>> +}
>> +
>> +void
>> +hypercall_disable(void)
>> +{
>> + return;
>> +}
>> +
>> +bool
>> +hyperv_identify_features(void)
>> +{
>> + struct hv_get_vp_registers_output result;
>> + vm_guest = VM_GUEST_HV;
>> +
>> + hv_get_vpreg_128(CPUID_LEAF_HV_FEATURES, &result);
>> + hyperv_features = result.as32.a;
>> + hv_get_vpreg_128(CPUID_LEAF_HV_IDENTITY, &result);
>> + hyperv_ver_major = result.as32.b >> 16;
>> + hv_get_vpreg_128(CPUID_LEAF_HV_RECOMMENDS, &result);
>> + hyperv_recommends = result.as32.a;
>> + return (true);
>> +}
>> +/*- SPDX-License-Identifier: BSD-2-Clause-FreeBSD
>> + * Copyright (c) 2009-2012,2016-2017, 2022 Microsoft Corp.
>> + * Copyright (c) 2012 NetApp Inc.
>> + * Copyright (c) 2012 Citrix Inc.
>> + * All rights reserved.
>> + *
>> + * Redistribution and use in source and binary forms, with or without
>> + * modification, are permitted provided that the following conditions
>> + * are met:
>> + * 1. Redistributions of source code must retain the above copyright
>> + * notice unmodified, this list of conditions, and the following
>> + * disclaimer.
>> + * 2. Redistributions in binary form must reproduce the above copyright
>> + * notice, this list of conditions and the following disclaimer in the
>> + * documentation and/or other materials provided with the distribution.
>> + *
>> + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
>> + * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
>> + * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
>> + * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
>> + * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
>> + * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
>> + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
>> + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>> + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
>> + * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
>> + */
>> +
>> +/**
>> + * Implements low-level interactions with Hyper-V/Azure
>> + */
>> +
>> +#include <sys/cdefs.h>
>> +__FBSDID("$FreeBSD$");
>> +
>> +#include <sys/param.h>
>> +#include <sys/systm.h>
>> +#include <sys/kernel.h>
>> +#include <sys/malloc.h>
>> +#include <sys/timetc.h>
>> +
>> +#include <vm/vm.h>
>> +#include <vm/pmap.h>
>> +#include <vm/vm_extern.h>
>> +#include <vm/vm_kern.h>
>> +
>> +#include <dev/hyperv/include/hyperv.h>
>> +#include <dev/hyperv/include/hyperv_busdma.h>
>> +#include <dev/hyperv/vmbus/aarch64/hyperv_machdep.h>
>> +#include <dev/hyperv/vmbus/aarch64/hyperv_reg.h>
>> +#include <dev/hyperv/vmbus/hyperv_var.h>
>> +
>> +void hyperv_init_tc(void);
>> +int hypercall_page_setup(vm_paddr_t);
>> +void hypercall_disable(void);
>> +bool hyperv_identify_features(void);
>> +
>> +u_int hyperv_ver_major;
>> +u_int hyperv_features;
>> +u_int hyperv_recommends;
>> +
>> +hyperv_tc64_t hyperv_tc64;
>> +
>> +void
>> +hyperv_init_tc(void)
>> +{
>> + hyperv_tc64 = NULL;
>> +}
>> +
>> +int
>> +hypercall_page_setup(vm_paddr_t hc)
>> +{
>> + return (0);
>> +}
>> +
>> +void
>> +hypercall_disable(void)
>> +{
>> + return;
>> +}
>> +
>> +bool
>> +hyperv_identify_features(void)
>> +{
>> + struct hv_get_vp_registers_output result;
>> + vm_guest = VM_GUEST_HV;
>> +
>> + hv_get_vpreg_128(CPUID_LEAF_HV_FEATURES, &result);
>> + hyperv_features = result.as32.a;
>> + hv_get_vpreg_128(CPUID_LEAF_HV_IDENTITY, &result);
>> + hyperv_ver_major = result.as32.b >> 16;
>> + hv_get_vpreg_128(CPUID_LEAF_HV_RECOMMENDS, &result);
>> + hyperv_recommends = result.as32.a;
>> + return (true);
>> +}
>> diff --git a/sys/dev/hyperv/vmbus/aarch64/hyperv_machdep.c b/sys/dev/hyperv/vmbus/aarch64/hyperv_machdep.c
>> new file mode 100644
>> index 000000000000..9bfe34044335
>> --- /dev/null
>> +++ b/sys/dev/hyperv/vmbus/aarch64/hyperv_machdep.c
>> @@ -0,0 +1,103 @@
>> +/*- SPDX-License-Identifier: BSD-2-Clause-FreeBSD
>> + *
>> + * Copyright (c) 2016-2017,2022 Microsoft Corp.
>> + *
>> + * Redistribution and use in source and binary forms, with or without
>> + * modification, are permitted provided that the following conditions
>> + * are met:
>> + * 1. Redistributions of source code must retain the above copyright
>> + * notice unmodified, this list of conditions, and the following
>> + * disclaimer.
>> + * 2. Redistributions in binary form must reproduce the above copyright
>> + * notice, this list of conditions and the following disclaimer in the
>> + * documentation and/or other materials provided with the distribution.
>> + *
>> + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
>> + * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
>> + * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
>> + * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
>> + * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
>> + * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
>> + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
>> + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>> + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
>> + * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
>> + */
>> +
>> +#include <sys/cdefs.h>
>> +__FBSDID("$FreeBSD$");
>> +
>> +#include <dev/hyperv/include/hyperv.h>
>> +#include <dev/hyperv/include/hyperv_busdma.h>
>> +#include <dev/hyperv/vmbus/aarch64/hyperv_machdep.h>
>> +#include <dev/hyperv/vmbus/aarch64/hyperv_reg.h>
>> +#include <dev/hyperv/vmbus/hyperv_var.h>
>> +#include <dev/psci/smccc.h>
>> +
>> +#define HVCALL_SET_VP_REGISTERS 0x0051
>> +#define HVCALL_GET_VP_REGISTERS 0x0050
>> +#define BIT(A) (1 << (A))
>> +#define HV_HYPERCALL_FAST_BIT BIT(16)
>> +#define BIT_ULL(a) (1ULL << (a))
>> +#define HV_HYPERCALL_REP_COMP_1 BIT_ULL(32)
>> +#define HV_PARTITION_ID_SELF ((u64)-1)
>> +#define HV_VP_INDEX_SELF ((u32)-2)
>> +#define HV_SMCCC_FUNC_NUMBER 1
>> +
>> +#define HV_FUNC_ID \
>> + SMCCC_FUNC_ID(SMCCC_YIELDING_CALL, SMCCC_64BIT_CALL, \
>> + SMCCC_VENDOR_HYP_SERVICE_CALLS, HV_SMCCC_FUNC_NUMBER)
>> +
>> +void
>> +arm_hv_set_vreg(u32 msr, u64 value)
>> +{
>> + arm_smccc_hvc(HV_FUNC_ID,
>> + HVCALL_SET_VP_REGISTERS | HV_HYPERCALL_FAST_BIT |
>> + HV_HYPERCALL_REP_COMP_1,
>> + HV_PARTITION_ID_SELF, HV_VP_INDEX_SELF, msr, 0, value, 0, NULL);
>> +}
>> +
>> +void
>> +hv_get_vpreg_128(u32 msr, struct hv_get_vp_registers_output *result)
>> +{
>> + struct arm_smccc_1_2_regs args;
>> + struct arm_smccc_1_2_regs res;
>> +
>> + args.a0 = HV_FUNC_ID;
>> + args.a1 = HVCALL_GET_VP_REGISTERS | HV_HYPERCALL_FAST_BIT |
>> + HV_HYPERCALL_REP_COMP_1;
>> + args.a2 = HV_PARTITION_ID_SELF;
>> + args.a3 = HV_VP_INDEX_SELF;
>> + args.a4 = msr;
>> +
>> + /*
>> + * Use the SMCCC 1.2 interface because the results are in registers
>> + * beyond X0-X3.
>> + */
>> + arm_smccc_1_2_hvc(&args, &res);
>> +
>> + result->as64.low = res.a6;
>> + result->as64.high = res.a7;
>> +}
>> +
>> +u64
>> +arm_hv_get_vreg(u32 msr)
>> +{
>> + struct hv_get_vp_registers_output output;
>> +
>> + hv_get_vpreg_128(msr, &output);
>> +
>> + return (output.as64.low);
>> +}
>> +
>> +uint64_t
>> +hypercall_md(volatile void *hc_addr, uint64_t in_val, uint64_t in_paddr,
>> + uint64_t out_paddr)
>> +{
>> + struct arm_smccc_res res;
>> +
>> + arm_smccc_hvc(HV_FUNC_ID, in_val, in_paddr, out_paddr, 0, 0, 0, 0,
>> + &res);
>> +
>> + return (res.a0);
>> +}
>> diff --git a/sys/dev/hyperv/vmbus/aarch64/hyperv_machdep.h b/sys/dev/hyperv/vmbus/aarch64/hyperv_machdep.h
>> new file mode 100644
>> index 000000000000..b017fdbc14c9
>> --- /dev/null
>> +++ b/sys/dev/hyperv/vmbus/aarch64/hyperv_machdep.h
>> @@ -0,0 +1,59 @@
>> +/*- SPDX-License-Identifier: BSD-2-Clause-FreeBSD
>> + *
>> + * Copyright (c) 2022 Microsoft Corp.
>> + *
>> + * Redistribution and use in source and binary forms, with or without
>> + * modification, are permitted provided that the following conditions
>> + * are met:
>> + * 1. Redistributions of source code must retain the above copyright
>> + * notice unmodified, this list of conditions, and the following
>> + * disclaimer.
>> + * 2. Redistributions in binary form must reproduce the above copyright
>> + * notice, this list of conditions and the following disclaimer in the
>> + * documentation and/or other materials provided with the distribution.
>> + *
>> + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
>> + * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
>> + * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
>> + * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
>> + * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
>> + * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
>> + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
>> + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>> + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
>> + * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
>> + *
>> + * $FreeBSD$
>> + */
>> +
>> +#ifndef _HYPERV_MACHDEP_H_
>> +#define _HYPERV_MACHDEP_H_
>> +
>> +#include <sys/param.h>
>> +
>> +typedef uint32_t u32;
>> +typedef uint64_t u64;
>> +struct hv_get_vp_registers_output {
>> + union {
>> + struct {
>> + u32 a;
>> + u32 b;
>> + u32 c;
>> + u32 d;
>> + } as32 __packed;
>> + struct {
>> + u64 low;
>> + u64 high;
>> + } as64 __packed;
>> + };
>> +};
>> +
>> +uint64_t hypercall_md(volatile void *hc_addr, u64 in_val,
>> + u64 in_paddr, u64 out_paddr);
>> +void hv_get_vpreg_128(u32, struct hv_get_vp_registers_output *);
>> +void arm_hv_set_vreg(u32 msr, u64 val);
>> +#define WRMSR(msr, val) arm_hv_set_vreg(msr, val)
>> +u64 arm_hv_get_vreg(u32 msr);
>> +#define RDMSR(msr) arm_hv_get_vreg(msr)
>> +
>> +#endif /* !_HYPERV_MACHDEP_H_ */
>> diff --git a/sys/dev/hyperv/vmbus/aarch64/hyperv_reg.h b/sys/dev/hyperv/vmbus/aarch64/hyperv_reg.h
>> new file mode 100644
>> index 000000000000..132b40768790
>> --- /dev/null
>> +++ b/sys/dev/hyperv/vmbus/aarch64/hyperv_reg.h
>> @@ -0,0 +1,386 @@
>> +/*- SPDX-License-Identifier: BSD-2-Clause-FreeBSD
>> + * Copyright (c) 2022 Microsoft Corp.
>> + * All rights reserved.
>> + *
>> + * Redistribution and use in source and binary forms, with or without
>> + * modification, are permitted provided that the following conditions
>> + * are met:
>> + * 1. Redistributions of source code must retain the above copyright
>> + * notice unmodified, this list of conditions, and the following
>> + * disclaimer.
>> + * 2. Redistributions in binary form must reproduce the above copyright
>> + * notice, this list of conditions and the following disclaimer in the
>> + * documentation and/or other materials provided with the distribution.
>> + *
>> + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
>> + * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
>> + * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
>> + * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
>> + * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
>> + * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
>> + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
>> + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>> + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
>> + * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
>> + *
>> + * $FreeBSD$
>> + */
>> +
>> +#ifndef _HYPERV_REG_H_
>> +#define _HYPERV_REG_H_
>> +
>> +#include <sys/param.h>
>> +#include <sys/systm.h>
>> +
>> +/*
>> + * Hyper-V Synthetic MSRs
>> + */
>> +
>> +#define MSR_HV_GUEST_OS_ID 0x00090002
>> +#define MSR_HV_GUESTID_BUILD_MASK 0xffffULL
>> +#define MSR_HV_GUESTID_VERSION_MASK 0x0000ffffffff0000ULL
>> +#define MSR_HV_GUESTID_VERSION_SHIFT 16
>> +#define MSR_HV_GUESTID_OSID_MASK 0x00ff000000000000ULL
>> +#define MSR_HV_GUESTID_OSID_SHIFT 48
>> +#define MSR_HV_GUESTID_OSTYPE_MASK 0x7f00000000000000ULL
>> +#define MSR_HV_GUESTID_OSTYPE_SHIFT 56
>> +#define MSR_HV_GUESTID_OPENSRC 0x8000000000000000ULL
>> +#define MSR_HV_GUESTID_OSTYPE_LINUX \
>> + ((0x01ULL << MSR_HV_GUESTID_OSTYPE_SHIFT) | MSR_HV_GUESTID_OPENSRC)
>> +#define MSR_HV_GUESTID_OSTYPE_FREEBSD \
>> + ((0x02ULL << MSR_HV_GUESTID_OSTYPE_SHIFT) | MSR_HV_GUESTID_OPENSRC)
>> +
>> +#define MSR_HV_HYPERCALL 0x40000001
>> +#define MSR_HV_HYPERCALL_ENABLE 0x0001ULL
>> +#define MSR_HV_HYPERCALL_RSVD_MASK 0x0ffeULL
>> +#define MSR_HV_HYPERCALL_PGSHIFT 12
>> +
>> +#define MSR_HV_VP_INDEX 0x00090003
>> +
>> +#define MSR_HV_REFERENCE_TSC 0x40000021
>> +#define MSR_HV_REFTSC_ENABLE 0x0001ULL
>> +#define MSR_HV_REFTSC_RSVD_MASK 0x0ffeULL
>> +#define MSR_HV_REFTSC_PGSHIFT 12
>> +
>> +#define MSR_HV_SCONTROL 0x000A0010
>> +#define MSR_HV_SCTRL_ENABLE 0x0001ULL
>> +#define MSR_HV_SCTRL_RSVD_MASK 0xfffffffffffffffeULL
>> +
>> +#define MSR_HV_SIEFP 0x000A0012
>> +#define MSR_HV_SIEFP_ENABLE 0x0001ULL
>> +#define MSR_HV_SIEFP_RSVD_MASK 0x0ffeULL
>> +#define MSR_HV_SIEFP_PGSHIFT 12
>> +
>> +#define MSR_HV_SIMP 0x000A0013
>> +#define MSR_HV_SIMP_ENABLE 0x0001ULL
>> +#define MSR_HV_SIMP_RSVD_MASK 0x0ffeULL
>> +#define MSR_HV_SIMP_PGSHIFT 12
>> +
>> +#define MSR_HV_EOM 0x000A0014
>> +
>> +#define MSR_HV_SINT0 0x000A0000
>> +#define MSR_HV_SINT_VECTOR_MASK 0x00ffULL
>> +#define MSR_HV_SINT_RSVD1_MASK 0xff00ULL
>> +#define MSR_HV_SINT_MASKED 0x00010000ULL
>> +#define MSR_HV_SINT_AUTOEOI 0x00000000ULL
>> +#define MSR_HV_SINT_RSVD2_MASK 0xfffffffffffc0000ULL
>> +#define MSR_HV_SINT_RSVD_MASK (MSR_HV_SINT_RSVD1_MASK | \
>> + MSR_HV_SINT_RSVD2_MASK)
>> +
>> +#define MSR_HV_STIMER0_CONFIG 0x400000b0
>> +#define MSR_HV_STIMER_CFG_ENABLE 0x0001ULL
>> +#define MSR_HV_STIMER_CFG_PERIODIC 0x0002ULL
>> +#define MSR_HV_STIMER_CFG_LAZY 0x0004ULL
>> +#define MSR_HV_STIMER_CFG_AUTOEN 0x0008ULL
>> +#define MSR_HV_STIMER_CFG_SINT_MASK 0x000f0000ULL
>> +#define MSR_HV_STIMER_CFG_SINT_SHIFT 16
>> +
>> +#define MSR_HV_STIMER0_COUNT 0x400000b1
>> +
>> +/*
>> + * CPUID leaves
>> + */
>> +
>> +#define CPUID_LEAF_HV_MAXLEAF 0x40000000
>> +
>> +#define CPUID_LEAF_HV_INTERFACE 0x40000001
>> +#define CPUID_HV_IFACE_HYPERV 0x31237648 /* HV#1 */
>> +
>> +#define CPUID_LEAF_HV_IDENTITY 0x00000100
>> +
>> +#define CPUID_LEAF_HV_FEATURES 0x00000200
>> +/* EAX: features include/hyperv.h CPUID_HV_MSR */
>> +/* ECX: power management features */
>> +#define CPUPM_HV_CSTATE_MASK 0x000f /* deepest C-state */
>> +#define CPUPM_HV_C3_HPET 0x0010 /* C3 requires HPET */
>> +#define CPUPM_HV_CSTATE(f) ((f) & CPUPM_HV_CSTATE_MASK)
>> +/* EDX: features3 */
>> +#define CPUID3_HV_MWAIT 0x0001 /* MWAIT */
>> +#define CPUID3_HV_XMM_HYPERCALL 0x0010 /* Hypercall input through
>> + * XMM regs */
>> +#define CPUID3_HV_GUEST_IDLE 0x0020 /* guest idle */
>> +#define CPUID3_HV_NUMA 0x0080 /* NUMA distance query */
>> +#define CPUID3_HV_TIME_FREQ 0x0100 /* timer frequency query
>> + * (TSC, LAPIC) */
>> +#define CPUID3_HV_MSR_CRASH 0x0400 /* MSRs for guest crash */
>> +
>> +#define CPUID_LEAF_HV_RECOMMENDS 0x00000201
>> +#define CPUID_LEAF_HV_LIMITS 0x40000005
>> +#define CPUID_LEAF_HV_HWFEATURES 0x40000006
>> +
>> +/*
>> + * Hyper-V Monitor Notification Facility
>> + */
>> +struct hyperv_mon_param {
>> + uint32_t mp_connid;
>> + uint16_t mp_evtflag_ofs;
>> + uint16_t mp_rsvd;
>> +} __packed;
>> +
>> +/*
>> + * Hyper-V message types
>> + */
>> +#define HYPERV_MSGTYPE_NONE 0
>> +#define HYPERV_MSGTYPE_CHANNEL 1
>> +#define HYPERV_MSGTYPE_TIMER_EXPIRED 0x80000010
>> +
>> +/*
>> + * Hypercall status codes
>> + */
>> +#define HYPERCALL_STATUS_SUCCESS 0x0000
>> +
>> +/*
>> + * Hypercall input values
>> + */
>> +#define HYPERCALL_POST_MESSAGE 0x005c
>> +#define HYPERCALL_SIGNAL_EVENT 0x005d
>> +
>> +/*
>> + * Hypercall input parameters
>> + */
>> +#define HYPERCALL_PARAM_ALIGN 8
>> +#if 0
>> +/*
>> + * XXX
>> + * <<Hypervisor Top Level Functional Specification 4.0b>> requires
>> + * input parameters size to be multiple of 8, however, many post
>> + * message input parameters do _not_ meet this requirement.
>> + */
>> +#define HYPERCALL_PARAM_SIZE_ALIGN 8
>> +#endif
>> +
>> +/*
>> + * HYPERCALL_POST_MESSAGE
>> + */
>> +#define HYPERCALL_POSTMSGIN_DSIZE_MAX 240
>> +#define HYPERCALL_POSTMSGIN_SIZE 256
>> +
>> +struct hypercall_postmsg_in {
>> + uint32_t hc_connid;
>> + uint32_t hc_rsvd;
>> + uint32_t hc_msgtype; /* HYPERV_MSGTYPE_ */
>> + uint32_t hc_dsize;
>> + uint8_t hc_data[HYPERCALL_POSTMSGIN_DSIZE_MAX];
>> +} __packed;
>> +CTASSERT(sizeof(struct hypercall_postmsg_in) == HYPERCALL_POSTMSGIN_SIZE);
>> +
>> +/*
>> + * HYPERCALL_SIGNAL_EVENT
>> + *
>> + * struct hyperv_mon_param.
>> + */
>> +
>> +#endif /* !_HYPERV_REG_H_ */
>> +/*- SPDX-License-Identifier: BSD-2-Clause-FreeBSD
>> + * Copyright (c) 2022 Microsoft Corp.
>> + * All rights reserved.
>> + *
>> + * Redistribution and use in source and binary forms, with or without
>> + * modification, are permitted provided that the following conditions
>> + * are met:
>> + * 1. Redistributions of source code must retain the above copyright
>> + * notice unmodified, this list of conditions, and the following
>> + * disclaimer.
>> + * 2. Redistributions in binary form must reproduce the above copyright
>> + * notice, this list of conditions and the following disclaimer in the
>> + * documentation and/or other materials provided with the distribution.
>> + *
>> + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
>> + * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
>> + * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
>> + * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
>> + * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
>> + * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
>> + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
>> + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>> + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
>> + * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
>> + *
>> + * $FreeBSD$
>> + */
>> +
>> +#ifndef _HYPERV_REG_H_
>> +#define _HYPERV_REG_H_
>> +
>> +#include <sys/param.h>
>> +#include <sys/systm.h>
>> +
>> +/*
>> + * Hyper-V Synthetic MSRs
>> + */
>> +
>> +#define MSR_HV_GUEST_OS_ID 0x00090002
>> +#define MSR_HV_GUESTID_BUILD_MASK 0xffffULL
>> +#define MSR_HV_GUESTID_VERSION_MASK 0x0000ffffffff0000ULL
>> +#define MSR_HV_GUESTID_VERSION_SHIFT 16
>> +#define MSR_HV_GUESTID_OSID_MASK 0x00ff000000000000ULL
>> +#define MSR_HV_GUESTID_OSID_SHIFT 48
>> +#define MSR_HV_GUESTID_OSTYPE_MASK 0x7f00000000000000ULL
>> +#define MSR_HV_GUESTID_OSTYPE_SHIFT 56
>> +#define MSR_HV_GUESTID_OPENSRC 0x8000000000000000ULL
>> +#define MSR_HV_GUESTID_OSTYPE_LINUX \
>> + ((0x01ULL << MSR_HV_GUESTID_OSTYPE_SHIFT) | MSR_HV_GUESTID_OPENSRC)
>> +#define MSR_HV_GUESTID_OSTYPE_FREEBSD \
>> + ((0x02ULL << MSR_HV_GUESTID_OSTYPE_SHIFT) | MSR_HV_GUESTID_OPENSRC)
>> +
>> +#define MSR_HV_HYPERCALL 0x40000001
>> +#define MSR_HV_HYPERCALL_ENABLE 0x0001ULL
>> +#define MSR_HV_HYPERCALL_RSVD_MASK 0x0ffeULL
>> +#define MSR_HV_HYPERCALL_PGSHIFT 12
>> +
>> +#define MSR_HV_VP_INDEX 0x00090003
>> +
>> +#define MSR_HV_REFERENCE_TSC 0x40000021
>> +#define MSR_HV_REFTSC_ENABLE 0x0001ULL
>> +#define MSR_HV_REFTSC_RSVD_MASK 0x0ffeULL
>> +#define MSR_HV_REFTSC_PGSHIFT 12
>> +
>> +#define MSR_HV_SCONTROL 0x000A0010
>> +#define MSR_HV_SCTRL_ENABLE 0x0001ULL
>> +#define MSR_HV_SCTRL_RSVD_MASK 0xfffffffffffffffeULL
>> +
>> +#define MSR_HV_SIEFP 0x000A0012
>> +#define MSR_HV_SIEFP_ENABLE 0x0001ULL
>> +#define MSR_HV_SIEFP_RSVD_MASK 0x0ffeULL
>> +#define MSR_HV_SIEFP_PGSHIFT 12
>> +
>> +#define MSR_HV_SIMP 0x000A0013
>> +#define MSR_HV_SIMP_ENABLE 0x0001ULL
>> +#define MSR_HV_SIMP_RSVD_MASK 0x0ffeULL
>> +#define MSR_HV_SIMP_PGSHIFT 12
>> +
>> +#define MSR_HV_EOM 0x000A0014
>> +
>> +#define MSR_HV_SINT0 0x000A0000
>> +#define MSR_HV_SINT_VECTOR_MASK 0x00ffULL
>> +#define MSR_HV_SINT_RSVD1_MASK 0xff00ULL
>> +#define MSR_HV_SINT_MASKED 0x00010000ULL
>> +#define MSR_HV_SINT_AUTOEOI 0x00000000ULL
>> +#define MSR_HV_SINT_RSVD2_MASK 0xfffffffffffc0000ULL
>> +#define MSR_HV_SINT_RSVD_MASK (MSR_HV_SINT_RSVD1_MASK | \
>> + MSR_HV_SINT_RSVD2_MASK)
>> +
>> +#define MSR_HV_STIMER0_CONFIG 0x400000b0
>> +#define MSR_HV_STIMER_CFG_ENABLE 0x0001ULL
>> +#define MSR_HV_STIMER_CFG_PERIODIC 0x0002ULL
>> +#define MSR_HV_STIMER_CFG_LAZY 0x0004ULL
>> +#define MSR_HV_STIMER_CFG_AUTOEN 0x0008ULL
>> +#define MSR_HV_STIMER_CFG_SINT_MASK 0x000f0000ULL
>> +#define MSR_HV_STIMER_CFG_SINT_SHIFT 16
>> +
>> +#define MSR_HV_STIMER0_COUNT 0x400000b1
>> +
>> +/*
>> + * CPUID leaves
>> + */
>> +
>> +#define CPUID_LEAF_HV_MAXLEAF 0x40000000
>> +
>> +#define CPUID_LEAF_HV_INTERFACE 0x40000001
>> +#define CPUID_HV_IFACE_HYPERV 0x31237648 /* HV#1 */
>> +
>> +#define CPUID_LEAF_HV_IDENTITY 0x00000100
>> +
>> +#define CPUID_LEAF_HV_FEATURES 0x00000200
>> +/* EAX: features include/hyperv.h CPUID_HV_MSR */
>> +/* ECX: power management features */
>> +#define CPUPM_HV_CSTATE_MASK 0x000f /* deepest C-state */
>> +#define CPUPM_HV_C3_HPET 0x0010 /* C3 requires HPET */
>> +#define CPUPM_HV_CSTATE(f) ((f) & CPUPM_HV_CSTATE_MASK)
>> +/* EDX: features3 */
>> +#define CPUID3_HV_MWAIT 0x0001 /* MWAIT */
>> +#define CPUID3_HV_XMM_HYPERCALL 0x0010 /* Hypercall input through
>> + * XMM regs */
>> +#define CPUID3_HV_GUEST_IDLE 0x0020 /* guest idle */
>> +#define CPUID3_HV_NUMA 0x0080 /* NUMA distance query */
>> +#define CPUID3_HV_TIME_FREQ 0x0100 /* timer frequency query
>> + * (TSC, LAPIC) */
>> +#define CPUID3_HV_MSR_CRASH 0x0400 /* MSRs for guest crash */
>> +
>> +#define CPUID_LEAF_HV_RECOMMENDS 0x00000201
>> +#define CPUID_LEAF_HV_LIMITS 0x40000005
>> +#define CPUID_LEAF_HV_HWFEATURES 0x40000006
>> +
>> +/*
>> + * Hyper-V Monitor Notification Facility
>> + */
>> +struct hyperv_mon_param {
>> + uint32_t mp_connid;
>> + uint16_t mp_evtflag_ofs;
>> + uint16_t mp_rsvd;
>> +} __packed;
>> +
>> +/*
>> + * Hyper-V message types
>> + */
>> +#define HYPERV_MSGTYPE_NONE 0
>> +#define HYPERV_MSGTYPE_CHANNEL 1
>> +#define HYPERV_MSGTYPE_TIMER_EXPIRED 0x80000010
>> +
>> +/*
>> + * Hypercall status codes
>> + */
>> +#define HYPERCALL_STATUS_SUCCESS 0x0000
>> +
>> +/*
>> + * Hypercall input values
>> + */
>> +#define HYPERCALL_POST_MESSAGE 0x005c
>> +#define HYPERCALL_SIGNAL_EVENT 0x005d
>> +
>> +/*
>> + * Hypercall input parameters
>> + */
>> +#define HYPERCALL_PARAM_ALIGN 8
>> +#if 0
>> +/*
>> + * XXX
>> + * <<Hypervisor Top Level Functional Specification 4.0b>> requires
>> + * input parameters size to be multiple of 8, however, many post
>> + * message input parameters do _not_ meet this requirement.
>> + */
>> +#define HYPERCALL_PARAM_SIZE_ALIGN 8
>> +#endif
>> +
>> +/*
>> + * HYPERCALL_POST_MESSAGE
>> + */
>> +#define HYPERCALL_POSTMSGIN_DSIZE_MAX 240
>> +#define HYPERCALL_POSTMSGIN_SIZE 256
>> +
>> +struct hypercall_postmsg_in {
>> + uint32_t hc_connid;
>> + uint32_t hc_rsvd;
>> + uint32_t hc_msgtype; /* HYPERV_MSGTYPE_ */
>> + uint32_t hc_dsize;
>> + uint8_t hc_data[HYPERCALL_POSTMSGIN_DSIZE_MAX];
>> +} __packed;
>> +CTASSERT(sizeof(struct hypercall_postmsg_in) == HYPERCALL_POSTMSGIN_SIZE);
>> +
>> +/*
>> + * HYPERCALL_SIGNAL_EVENT
>> + *
>> + * struct hyperv_mon_param.
>> + */
>> +
>> +#endif /* !_HYPERV_REG_H_ */
>> diff --git a/sys/dev/hyperv/vmbus/aarch64/vmbus_aarch64.c b/sys/dev/hyperv/vmbus/aarch64/vmbus_aarch64.c
>> new file mode 100644
>> index 000000000000..ba4d8c9bc36e
>> --- /dev/null
>> +++ b/sys/dev/hyperv/vmbus/aarch64/vmbus_aarch64.c
>> @@ -0,0 +1,314 @@
>> +/*- SPDX-License-Identifier: BSD-2-Clause-FreeBSD
>> + * Copyright (c) 2009-2012,2016-2017, 2022 Microsoft Corp.
>> + * Copyright (c) 2012 NetApp Inc.
>> + * Copyright (c) 2012 Citrix Inc.
>> + * All rights reserved.
>> + *
>> + * Redistribution and use in source and binary forms, with or without
>> + * modification, are permitted provided that the following conditions
>> + * are met:
>> + * 1. Redistributions of source code must retain the above copyright
>> + * notice unmodified, this list of conditions, and the following
>> + * disclaimer.
>> + * 2. Redistributions in binary form must reproduce the above copyright
>> + * notice, this list of conditions and the following disclaimer in the
>> + * documentation and/or other materials provided with the distribution.
>> + *
>> + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
>> + * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
>> + * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
>> + * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
>> + * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
>> + * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
>> + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
>> + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>> + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
>> + * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
>> + */
>> +
>> +/*
>> + * VM Bus Driver Implementation
>> + */
>> +
>> +#include <sys/cdefs.h>
>> +__FBSDID("$FreeBSD$");
>> +
>> +#include <sys/param.h>
>> +#include <sys/bus.h>
>> +#include <sys/kernel.h>
>> +#include <sys/linker.h>
>> +#include <sys/lock.h>
>> +#include <sys/malloc.h>
>> +#include <sys/module.h>
>> +#include <sys/mutex.h>
>> +#include <sys/sbuf.h>
>> +#include <sys/smp.h>
>> +#include <sys/sysctl.h>
>> +#include <sys/systm.h>
>> +#include <sys/taskqueue.h>
>> +
>> +#include <vm/vm.h>
>> +#include <vm/vm_param.h>
>> +#include <vm/pmap.h>
>> +
>> +#include <machine/bus.h>
>> +#include <machine/metadata.h>
>> +#include <machine/md_var.h>
>> +#include <machine/resource.h>
>> +#include <contrib/dev/acpica/include/acpi.h>
>> +#include <dev/acpica/acpivar.h>
>> +
>> +#include <dev/hyperv/include/hyperv.h>
>> +#include <dev/hyperv/include/vmbus_xact.h>
>> +#include <dev/hyperv/vmbus/hyperv_var.h>
>> +#include <dev/hyperv/vmbus/vmbus_reg.h>
>> +#include <dev/hyperv/vmbus/vmbus_var.h>
>> +#include <dev/hyperv/vmbus/vmbus_chanvar.h>
>> +#include <dev/hyperv/vmbus/aarch64/hyperv_machdep.h>
>> +#include <dev/hyperv/vmbus/aarch64/hyperv_reg.h>
>> +#include "acpi_if.h"
>> +#include "pcib_if.h"
>> +#include "vmbus_if.h"
>> +
>> +static int vmbus_handle_intr_new(void *);
>> +
>> +void vmbus_handle_timer_intr1(struct vmbus_message *msg_base,
>> + struct trapframe *frame);
>> +void vmbus_synic_setup1(void *xsc);
>> +void vmbus_synic_teardown1(void);
>> +int vmbus_setup_intr1(struct vmbus_softc *sc);
>> +void vmbus_intr_teardown1(struct vmbus_softc *sc);
>> +
>> +void
>> +vmbus_handle_timer_intr1(struct vmbus_message *msg_base,
>> + struct trapframe *frame)
>> +{
>> + // do nothing for arm64, as we are using generic timer
>> + return;
>> +}
>> +
>> +static int
>> +vmbus_handle_intr_new(void *arg)
>> +{
>> + vmbus_handle_intr(NULL);
>> + return (FILTER_HANDLED);
>> +}
>> +
>> +void
>> +vmbus_synic_setup1(void *xsc)
>> +{
>> + return;
>> +}
>> +
>> +void
>> +vmbus_synic_teardown1(void)
>> +{
>> + return;
>> +}
>> +
>> +int
>> +vmbus_setup_intr1(struct vmbus_softc *sc)
>> +{
>> + int err;
>> + struct intr_map_data_acpi *irq_data;
>> +
>> + sc->ires = bus_alloc_resource_any(device_get_parent(sc->vmbus_dev),
>> + SYS_RES_IRQ, &sc->vector, RF_ACTIVE | RF_SHAREABLE);
>> + if (sc->ires == NULL) {
>> + device_printf(sc->vmbus_dev, "bus_alloc_resouce_any failed\n");
>> + return (ENXIO);
>> + } else {
>> + device_printf(sc->vmbus_dev, "irq 0x%lx, vector %d end 0x%lx\n",
>> + (uint64_t)rman_get_start(sc->ires), sc->vector,
>> + (uint64_t)rman_get_end(sc->ires));
>> + }
>> + err = bus_setup_intr(sc->vmbus_dev, sc->ires, INTR_TYPE_MISC,
>> + vmbus_handle_intr_new, NULL, sc, &sc->icookie);
>> + if (err) {
>> + device_printf(sc->vmbus_dev, "failed to setup IRQ %d\n", err);
>> + return (err);
>> + }
>> + irq_data = (struct intr_map_data_acpi *)rman_get_virtual(sc->ires);
>> + device_printf(sc->vmbus_dev, "the irq %u\n", irq_data->irq);
>> + sc->vmbus_idtvec = irq_data->irq;
>> + return 0;
>> +}
>> +
>> +void
>> +vmbus_intr_teardown1(struct vmbus_softc *sc)
>> +{
>> + int cpu;
>> +
>> + sc->vmbus_idtvec = -1;
>> + bus_teardown_intr(sc->vmbus_dev, sc->ires, sc->icookie);
>> +
>> + CPU_FOREACH(cpu) {
>> + if (VMBUS_PCPU_GET(sc, event_tq, cpu) != NULL) {
>> + taskqueue_free(VMBUS_PCPU_GET(sc, event_tq, cpu));
>> + VMBUS_PCPU_GET(sc, event_tq, cpu) = NULL;
>> + }
>> + if (VMBUS_PCPU_GET(sc, message_tq, cpu) != NULL) {
>> + taskqueue_drain(VMBUS_PCPU_GET(sc, message_tq, cpu),
>> + VMBUS_PCPU_PTR(sc, message_task, cpu));
>> + taskqueue_free(VMBUS_PCPU_GET(sc, message_tq, cpu));
>> + VMBUS_PCPU_GET(sc, message_tq, cpu) = NULL;
>> + }
>> + }
>> +}
>> +/*- SPDX-License-Identifier: BSD-2-Clause-FreeBSD
>> + * Copyright (c) 2009-2012,2016-2017, 2022 Microsoft Corp.
>> + * Copyright (c) 2012 NetApp Inc.
>> + * Copyright (c) 2012 Citrix Inc.
>> + * All rights reserved.
>> + *
>> + * Redistribution and use in source and binary forms, with or without
>> + * modification, are permitted provided that the following conditions
>> + * are met:
>> + * 1. Redistributions of source code must retain the above copyright
>> + * notice unmodified, this list of conditions, and the following
>> + * disclaimer.
>> + * 2. Redistributions in binary form must reproduce the above copyright
>> + * notice, this list of conditions and the following disclaimer in the
>> + * documentation and/or other materials provided with the distribution.
>> + *
>> + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
>> + * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
>> + * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
>> + * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
>> + * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
>> + * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
>> + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
>> + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>> + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
>> + * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
>> + */
>> +
>> +/*
>> + * VM Bus Driver Implementation
>> + */
>> +
>> *** 125 LINES SKIPPED ***
>>