From nobody Mon May 15 13:54:06 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QKgpK5s3Rz4Bdj9 for ; Mon, 15 May 2023 13:54:17 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QKgpK5Jy1z4Xqm for ; Mon, 15 May 2023 13:54:17 +0000 (UTC) (envelope-from kevans@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1684158857; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CO0p/mYi5U7MIErLO8ZnwiFexF2SzApxXRtoLNpWcdw=; b=pOMwYKJZIDHj1sgAdA6yfWfTgckMxucbJ+87KU87v87lEOzjJuycFaOxRuW0i8o+u/F0Ku /Jo2yk/t5kr8+O66xfZEZGMrLoXJCxOwDnFkr6MLSEaC6w+O8qAF7O3vNKkTgA4BipSIAF MM868Qb0UyIweVFWcqAifxnuUo719P09yemPApauOb4lX6juuvxNiM0FvN+/uajTytBDKl XRHMDT1KFFVDUTewmIhM3xO93MeylWn5J7MmezQ3L0V0sGV7X6r2bg5wb2Il+j8gFFvC/1 fuXpQ6Nb0DSi7H5CLf1j6ihj3QIId04Ud/r1DXi76eoYuzmXNfVX64/M7Kqxfw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1684158857; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CO0p/mYi5U7MIErLO8ZnwiFexF2SzApxXRtoLNpWcdw=; b=PashcELtO2yyI3wgHl/4nS0SsfvZS/q4K81il9OrEmKAiiJ+92jjmvCyCSh/lojj6Y4DtU QWI0R+51k1GhTGq59A6OHwOAHDHVa7TEui4gE10eyRIwyUFGAowb3xYhXLBq5xXY4N5eCs TxaS517+bhy6DwmB7YKxw4hX3XLyo5f9hrU9oKHCldeuRrDaNcOOGCAL9LqAUTcmomFCqo YOfidDPRCDHCvavl/bUjdtHJBWUXXe/Xn35yjsLMlWdFn2RuK3vIfOuiIIjSqqBSx9tPyz QBbNPUI/HfUet7hqKAPqKqE1q0DoHURJ6qGnalCm/qOMKFQwF+OmSxU669qB4g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1684158857; a=rsa-sha256; cv=none; b=BER3QllgTrXA6t38Wo6slZi8bO1epPDJxuFzbJblRXLErUsk7CTJCuLJMvGudyVBbzOWNn mtFizuU+Md20J8KTtSWtQKGlKAanXagAlyqO0YiqNDXx/W0r+IPDpQ2ABQBlqI9+Otlk9K VACMMAy0UdzudnLelNk+4f3k3ZR1H7m0i1ln+emFaSldkjNmaP7pt4l0PmiA/qlcXHxqBA 6yNOYB4gUKVioFNlQnI5Hpr7HRyBnh3L6XFqox2m0ge7MPcOUvo07lxyeMi7dXBms7iYo7 avsnjNqByBlESsGTpKN7JJcrlfhS5SNmWfTZaGf58eU06QnAnep05x6fKYo10A== Received: from mail-qk1-f171.google.com (mail-qk1-f171.google.com [209.85.222.171]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) (Authenticated sender: kevans) by smtp.freebsd.org (Postfix) with ESMTPSA id 4QKgpK4Ff9z10SL for ; Mon, 15 May 2023 13:54:17 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: by mail-qk1-f171.google.com with SMTP id af79cd13be357-759413d99afso414714285a.1 for ; Mon, 15 May 2023 06:54:17 -0700 (PDT) X-Gm-Message-State: AC+VfDyuLoCKjMKDb3nhbgEOliQb99hFOPp7NJTo4e7bi1tpYkUSY6+d /RqbmnNlN1gvXQqrXNcOy7eTWHVjhrLD4Mj99fE= X-Google-Smtp-Source: ACHHUZ78W0pq2XTuLsCBdhV0BiUarTzMRRswBhIHbNCI7+26NdFZsUF80C5Bt/J7KlWdAgAZ3Uywp433AIcKkxfmXgI= X-Received: by 2002:ad4:5bab:0:b0:621:5823:cf1c with SMTP id 11-20020ad45bab000000b006215823cf1cmr25069762qvq.14.1684158857193; Mon, 15 May 2023 06:54:17 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Kyle Evans Date: Mon, 15 May 2023 08:54:06 -0500 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [EXTERNAL] Re: enabling same PPI interrupt to all CPU in ARM64 SMP To: Souradeep Chakrabarti Cc: Wei Hu , "freebsd-hackers@FreeBSD.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-ThisMailContainsUnwantedMimeParts: N On Mon, May 15, 2023 at 2:41=E2=80=AFAM Souradeep Chakrabarti wrote: > > > > > >-----Original Message----- > >From: Kyle Evans > >Sent: Monday, May 15, 2023 10:36 AM > >To: Souradeep Chakrabarti > >Cc: Wei Hu ; freebsd-hackers@FreeBSD.org > >Subject: Re: [EXTERNAL] Re: enabling same PPI interrupt to all CPU in AR= M64 SMP > > > >On Sun, May 14, 2023 at 9:59=E2=80=AFAM Souradeep Chakrabarti > > wrote: > >> > >> > >> > >> > >> >-----Original Message----- > >> >From: Kyle Evans > >> >Sent: Friday, May 12, 2023 10:40 PM > >> >To: Souradeep Chakrabarti > >> >Cc: Wei Hu ; freebsd-hackers@FreeBSD.org > >> >Subject: [EXTERNAL] Re: enabling same PPI interrupt to all CPU in > >> >ARM64 SMP > >> > > >> >On Fri, May 12, 2023 at 9:51=E2=80=AFAM Souradeep Chakrabarti > >> > wrote: > >> >> > >> >> > >> >> > >> >> > >> >> >-----Original Message----- > >> >> >From: Souradeep Chakrabarti > >> >> >Sent: Monday, May 8, 2023 6:39 PM > >> >> >To: Kyle Evans > >> >> >Cc: Wei Hu ; freebsd-hackers@FreeBSD.org > >> >> >Subject: enabling same PPI interrupt to all CPU in ARM64 SMP > >> >> > > >> >> >Hi , > >> >> > > >> >> >While using SMP in ARM64 Hyper-V we are getting stuck in boot if > >> >> >there is a interrupt for VMBus coming to CPU1 and VMBus interrupt > >> >> >handler is not getting that interrupt. > >> >> > > >> >> >In ARM64 Hyper-V we are using IRQ18 for VMBus and it is a PPI inte= rrupt. > >> >> > > >> >> >But Hypev-V host sends interrupt to this IRQ 18 for both CPU0 and > >> >> >CPU1 in 2CPU system. > >> >> >This is based on the corresponding VMBus channel which assigned wi= th the > >CPU. > >> >> > > >> >> >Now VMBus ISR is getting the interrupt in CPU0 but not getting fro= m CPU1. > >> >> >Any idea, how we can use the same PPI 18 for all the CPU cores? > >> >> > > >> >> >Any help will be appreciated, as this is blocking the enablement > >> >> >of FreeBSD in Azure ARM64. > >> >> [Souradeep] > >> >> Can someone please help me it. > >> >> > >> > > >> >Looking at least at the GIC implementation, it looks like this is a k= nown limitation: > >> > > >> > 875 /* > >> > 876 * XXX - In case that per CPU interrupt is going to be > >> >enabled in time > >> > 877 * when SMP is already started, we need some IPI > >> >call which > >> > 878 * enables it on others CPUs. Further, it's more > >> >complicated as > >> > 879 * pic_enable_source() and pic_disable_source() > >> >should act on > >> > 880 * per CPU basis only. Thus, it should be solved > >> >here somehow. > >> > 881 */ > >> > 882 if (isrc->isrc_flags & INTR_ISRCF_PPI) > >> > 883 CPU_SET(PCPU_GET(cpuid), &isrc->isrc_cpu); > >> > > >> >I think we need something /like/ this: > >> >https://peo/ > >> > >>ple.fr%2F&data=3D05%7C01%7Cschakrabarti%40microsoft.com%7C39d7c50849b > >64 > >> > >>eaf70b608db55020fae%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0% > >7C6381 > >> > >>97239615544113%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLC > >JQIjoiV2 > >> > >>luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3DdXVvnI > >HhN9 > >> >mdwiPkSJKwMyEKYi5SyGOuta5zCZ1ysCQ%3D&reserved=3D0 > >> > >>eebsd.org%2F~kevans%2Fppi.diff&data=3D05%7C01%7Cschakrabarti%40microsof > >> >t.c > >om%7Cc5c3d254b9d841e9ae9b08db530bb3d2%7C72f988bf86f141af91ab2d7c > >> > >>d011db47%7C1%7C0%7C638195082027744706%7CUnknown%7CTWFpbGZsb > >3 > >> > >>d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D > >> > >>%7C3000%7C%7C%7C&sdata=3DSwoR2vHxh6QQhwpOgFcQ9378nDVovhdEKEXoE > >o > >> >gPKsc%3D&reserved=3D0, though it still has the caveat that PPIs > >> >effectively cannot be fully setup before SI_SUB_SMP. > >> >So, it's likely almost a NOP for existing platforms (will emit a > >> >warning with bootverbose for armv8 timers) but might do the trick for= you. > >> [Souradeep] > >> Thanks for the change but it did not solve the problem. Still the > >> interrupt handler vmbus_handle_intr(struct trapframe *trap_frame), is = not > >getting called for the CPU 1. > >> It is only getting called for CPU 0 all the time in ARM64 but in x86 > >> it is getting called for both CPU1 and CPU0. > > > >Interesting! I do see one problem with the patch (and some cosmetic > >issues): we really need to take the gic_mtx in gic_v3_setup_periph() rig= ht up front > >because CPU_SET() won't necessarily be atomic. That's not the problem, t= hough for > >other reasons, but also because... > > > >> From DDB I have collected this data in arm64. irq18 is for vmbus. > >> db> show irqs > >> ... > >> irq18 : cpu 03 cnt 411 > >> .... > >> > > > >That would seem to indicate that both CPUs have set it up, but it occurs= to me that > >enable_intr also needs the same treatment. Let's wipe gic_v3.c back to a= clean slate > >and try a v2 of the patch: > >https://people.fr/ > >eebsd.org%2F~kevans%2Fppi- > >v2.diff&data=3D05%7C01%7Cschakrabarti%40microsoft.com%7C39d7c50849b64ea > >f70b608db55020fae%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6 > >38197239615544113%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAi > >LCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdat > >a=3DQLDm8X1yuuGOy2OeeYxsnF%2FNeuceAogw5161U7ZH6%2Bs%3D&reserved=3D0 > > > >For now we just pretend that we won't be disabling any PPIs as a proof-o= f-concept. > > > [Souradeep] > This is causing panic during boot: > pmu0: MADT: cpu 0 (mpidr 0) irq 0 level-triggered > pmu0: MADT: cpu 1 (mpidr 1) irq 0 level-triggered > panic: gic_v3_enable_intr_impl: Unsupported IRQ 0 > cpuid =3D 0 > time =3D 1 > KDB: stack backtrace: > db_trace_self() at db_trace_self > db_trace_self_wrapper() at db_trace_self_wrapper+0x30 > vpanic() at vpanic+0x13c > panic() at panic+0x44 > gic_v3_enable_intr_impl() at gic_v3_enable_intr_impl+0xec > intr_setup_irq() at intr_setup_irq+0x368 > bus_setup_intr() at bus_setup_intr+0x94 > pmu_attach() at pmu_attach+0x64 > pmu_acpi_attach() at pmu_acpi_attach+0x94 > device_attach() at device_attach+0x3f8 > device_probe_and_attach() at device_probe_and_attach+0x7c > bus_generic_new_pass() at bus_generic_new_pass+0xfc > bus_generic_new_pass() at bus_generic_new_pass+0xac > bus_generic_new_pass() at bus_generic_new_pass+0xac > bus_set_pass() at bus_set_pass+0x4c > mi_startup() at mi_startup+0x1fc > virtdone() at virtdone+0x70 > KDB: enter: panic > [ thread pid 0 tid 100000 ] > Stopped at kdb_enter+0x44: str xzr, [x19, #3328] > Details of the log I have pasted here: https://pastebin.com/eQN5RntD I think you fetched a broken version of the patch -- there was maybe an hour where I kept replacing it because I found a problem and fixed it, but created another one to fix that I discovered on some hardware I tested on. I've re-uploaded the correct diff at https://people.freebsd.org/~kevans/ppi-v3.diff, which can't hit the panic because irq <=3D GIC_LAST_PPI all return early (the version you have, the first branch in gic_v3_enable_intr_impl was probably an incorrect `if (isrc->isrc_flags & INTR_ISRCF_PPI)` Thanks, Kyle Evans