From nobody Mon May 15 05:05:40 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QKS4c4hzwz4B8NQ for ; Mon, 15 May 2023 05:05:52 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QKS4c4GSyz3CNs for ; Mon, 15 May 2023 05:05:52 +0000 (UTC) (envelope-from kevans@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1684127152; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rJS7EqRUXtx5n3iiH/x4xds+s3HOpC5iQfuEl+iCMPA=; b=K8R3vSJ+pa5AV+teXjs9tiZy1giaYzW7HJTCNx3LYwSmSni7A5xXn+yO/3Y7kY9pshAsNI mdad1Bqn2kUVB5XTN0RYfeW/j601WOUWMcRPEHOCN161MLfv33m8GRbCN6ob87CH3Ie4pC HKuKpTqIggbq73bE7UPPFszLx8KUpdj/Fv+82hENFDTypQ3oAt3COxkXIDVV6j3g5KGcPC W1J3vEl7WUfM1d1ra/974SKJuPkl+kdE5P37S7SrsQZUxbGXdBH4IA+XpCPOGBr2irUq1n etODrHbrtdcvJ0f+eu8NYlirBdQGqGx/C0fRi30bg19Dk7g67RSkmsBBh6oT4g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1684127152; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rJS7EqRUXtx5n3iiH/x4xds+s3HOpC5iQfuEl+iCMPA=; b=mb+yzL3FlgxUIW1CC+H8MseY7TdD+UEzZ2V2+fwnom4S/a9bkjJO9+0CD51DnlwFUivNCe ZVXu0wELs4BuEifQKpZUBa9qWQ8BVog5WrpMGJQO4xxLiuWIxnAYnVpUy0+Q3DWQd553aN GiKLKoWWF4bsFO3q5HXEXvclXcWntgGRz3Yohsxegbpa+YXGUzr212narlVqxkV7mzI1AE qSu4sW2bVXj1sTlCab4Gk3XemuqEzi1wlYpHyEwOWy9j74K75HKxAIZkUH5WEix3n476kx CTCs3MtxTX/w/j/VNAOBE0hnYyvVtNlfQl2JIeKj2Vj4nv120gvwMPR2hPVlkg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1684127152; a=rsa-sha256; cv=none; b=XLOjMLDwxq3v8+/oTUgVI/35Zt7hfKmP+HG6pBh5MDss+SY7FRPs0ZvrGji70l671AuTLO yaX4vOwcO1hlMXOk7xRU8roxQzjBc0mxsVvAo6PHmjqP+vPuDJpwjpcXPKN5mDG/JwQuM9 umW+Yh3f85AS46PtkvxZAqND7up43T2H19/Xd20wymzzVmRXIcO79R+zBfYXBasfqVjX7h JQ2+ml9O1UImhDPcI08saUbJOkhGaqPhlt4xARjYdhRhmljOEdafnDKjJFgiyowm4vF1Hn 81HzzsxK5C6o2BH765jxIcBbcLpGQAi1DDolpTvQtlMpA86S82w6l7+Ce+at6Q== Received: from mail-qk1-f171.google.com (mail-qk1-f171.google.com [209.85.222.171]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) (Authenticated sender: kevans) by smtp.freebsd.org (Postfix) with ESMTPSA id 4QKS4c371ZzmwM for ; Mon, 15 May 2023 05:05:52 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: by mail-qk1-f171.google.com with SMTP id af79cd13be357-7577ef2fa31so2535103385a.0 for ; Sun, 14 May 2023 22:05:52 -0700 (PDT) X-Gm-Message-State: AC+VfDwSIDGVo24rCvbjRkd1mL6Z1ZPVwxkgNALhHLU2qSbLZDWYC9W5 vUwemFwS9tvAPYMgPRdKfKs/jJiZxgahiCJIjgk= X-Google-Smtp-Source: ACHHUZ4RwJ6FDGlk4rohLkSo08PxDR5Mt9qciZbBhxn8j9lmtQauKowCn/uNMtT2zKvB7B6Kfb+DC0KO7HuBVB/vces= X-Received: by 2002:a05:6214:c46:b0:621:53d1:3c69 with SMTP id r6-20020a0562140c4600b0062153d13c69mr25469061qvj.20.1684127151535; Sun, 14 May 2023 22:05:51 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Kyle Evans Date: Mon, 15 May 2023 00:05:40 -0500 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [EXTERNAL] Re: enabling same PPI interrupt to all CPU in ARM64 SMP To: Souradeep Chakrabarti Cc: Wei Hu , "freebsd-hackers@FreeBSD.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-ThisMailContainsUnwantedMimeParts: N On Sun, May 14, 2023 at 9:59=E2=80=AFAM Souradeep Chakrabarti wrote: > > > > > >-----Original Message----- > >From: Kyle Evans > >Sent: Friday, May 12, 2023 10:40 PM > >To: Souradeep Chakrabarti > >Cc: Wei Hu ; freebsd-hackers@FreeBSD.org > >Subject: [EXTERNAL] Re: enabling same PPI interrupt to all CPU in ARM64 = SMP > > > >On Fri, May 12, 2023 at 9:51=E2=80=AFAM Souradeep Chakrabarti > > wrote: > >> > >> > >> > >> > >> >-----Original Message----- > >> >From: Souradeep Chakrabarti > >> >Sent: Monday, May 8, 2023 6:39 PM > >> >To: Kyle Evans > >> >Cc: Wei Hu ; freebsd-hackers@FreeBSD.org > >> >Subject: enabling same PPI interrupt to all CPU in ARM64 SMP > >> > > >> >Hi , > >> > > >> >While using SMP in ARM64 Hyper-V we are getting stuck in boot if > >> >there is a interrupt for VMBus coming to CPU1 and VMBus interrupt > >> >handler is not getting that interrupt. > >> > > >> >In ARM64 Hyper-V we are using IRQ18 for VMBus and it is a PPI interru= pt. > >> > > >> >But Hypev-V host sends interrupt to this IRQ 18 for both CPU0 and > >> >CPU1 in 2CPU system. > >> >This is based on the corresponding VMBus channel which assigned with = the CPU. > >> > > >> >Now VMBus ISR is getting the interrupt in CPU0 but not getting from C= PU1. > >> >Any idea, how we can use the same PPI 18 for all the CPU cores? > >> > > >> >Any help will be appreciated, as this is blocking the enablement of > >> >FreeBSD in Azure ARM64. > >> [Souradeep] > >> Can someone please help me it. > >> > > > >Looking at least at the GIC implementation, it looks like this is a know= n limitation: > > > > 875 /* > > 876 * XXX - In case that per CPU interrupt is going to be > >enabled in time > > 877 * when SMP is already started, we need some IPI > >call which > > 878 * enables it on others CPUs. Further, it's more > >complicated as > > 879 * pic_enable_source() and pic_disable_source() > >should act on > > 880 * per CPU basis only. Thus, it should be solved > >here somehow. > > 881 */ > > 882 if (isrc->isrc_flags & INTR_ISRCF_PPI) > > 883 CPU_SET(PCPU_GET(cpuid), &isrc->isrc_cpu); > > > >I think we need something /like/ this: > >https://people.fr/ > >eebsd.org%2F~kevans%2Fppi.diff&data=3D05%7C01%7Cschakrabarti%40microsoft= .c > >om%7Cc5c3d254b9d841e9ae9b08db530bb3d2%7C72f988bf86f141af91ab2d7c > >d011db47%7C1%7C0%7C638195082027744706%7CUnknown%7CTWFpbGZsb3 > >d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D > >%7C3000%7C%7C%7C&sdata=3DSwoR2vHxh6QQhwpOgFcQ9378nDVovhdEKEXoEo > >gPKsc%3D&reserved=3D0, though it still has the caveat that PPIs effectiv= ely cannot be > >fully setup before SI_SUB_SMP. > >So, it's likely almost a NOP for existing platforms (will emit a warning= with > >bootverbose for armv8 timers) but might do the trick for you. > [Souradeep] > Thanks for the change but it did not solve the problem. Still the interru= pt handler > vmbus_handle_intr(struct trapframe *trap_frame), is not getting called fo= r the CPU 1. > It is only getting called for CPU 0 all the time in ARM64 but in x86 it i= s getting called > for both CPU1 and CPU0. Interesting! I do see one problem with the patch (and some cosmetic issues): we really need to take the gic_mtx in gic_v3_setup_periph() right up front because CPU_SET() won't necessarily be atomic. That's not the problem, though for other reasons, but also because... > From DDB I have collected this data in arm64. irq18 is for vmbus. > db> show irqs > ... > irq18 : cpu 03 cnt 411 > .... > That would seem to indicate that both CPUs have set it up, but it occurs to me that enable_intr also needs the same treatment. Let's wipe gic_v3.c back to a clean slate and try a v2 of the patch: https://people.freebsd.org/~kevans/ppi-v2.diff For now we just pretend that we won't be disabling any PPIs as a proof-of-concept. Thanks, Kyle Evans