From nobody Sat Sep 05 04:15:15 2026 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4hcKkC4W2Lz6qrdg for ; Sat, 05 Sep 2026 04:15:15 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "YR2" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4hcKkC2Rg9z3bVd for ; Sat, 05 Sep 2026 04:15:15 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1788581715; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=rHeAoMuTenAs9YBrC+LO/41yW9JQ3ZJMlkp1hL2PIvU=; b=EZrgnC7OGpQgUgAyhu8Of/OtD9s7OsYHhbj/uewJw0DMLa1kD1U9/1qtRuDlXmD7AniN2T j4gmzJhVJM4l7jJvYtVU7CvrCsVTbD1sVFleBKgU0jSrPaMQXLwslaXs+fnxQaweEGeCdi h+Mk9jfP1AxrhsOAVEyPqMJ7s/VwwmjhbUlsc7MP/3eLcui0/TvKq0Hp2UX5QuUB24w7ZC NmRm5+66t5Ae/q11tdV2Ot9QZls37JbTVEcF3qDSI/ypzDLzQ4H/LMRX/t/WRZDDDH1PS4 O8RncXQjDMPM3Fl7go5FJz1kP2mSGJLuhYmnu5T0DgUFSAjd/IFSnf3ONgTo0A== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1788581715; a=rsa-sha256; cv=none; b=lwLl3WcOHDP9+jMMgd09AFxIQ4lrdNvIc68u0KxdNUOyQD/dAxtzXLOM+sWJVNpETKef2T TlKl0vP1phfYu74hBHGFsmK1NP1+vWNiiMpd03xj+aQdZxmOHPcsUtDQHfsDe+36icdoEc MkF8c8L7sxDpl365ES5D0Ov0HZ+JshYu7o3mJMtldiedAaHJF7qZ+ocpsF0V6rlcnKWV3Z G9ZuQ6rxkSGTnSTAcfBSqAmEINfIFzX3Ah+yIk7CuapqjovGHqF8EHrU1zx3bPYPMiRBIa wHUjiGIE/qZg35uQ2hXzVJvMJ97bc6kBENp08EdbBZbus0WmAU30CLBYuBFMZA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1788581715; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=rHeAoMuTenAs9YBrC+LO/41yW9JQ3ZJMlkp1hL2PIvU=; b=Eb7tKAkrmpGkeLGA99TvUILEzhwh4GgJTIVTlH6KgH/YrU9cAi3muEzQZcuM88nE7xdhrC rojxGN/nkojcO8Fcitd+Poq4Yh40ea1GB241wEZVF3csVo6fRPL3Nz95/L/Ds7O0SVhZ4Q mr13KgHGz/y9VYjtO9BmlJatGR4S2PQyZN0C7FpJwMjriEQh4B3TRM1f0PSLTJPKTVvfnB Q0sxRsh3vT1+GS+ibBsjfsbTRy+pj2Pw3//U7fCzNdYqYwuM+lP/9l2KL4uKvi3AxH32CC 0EpKxdFtm30VINV6Zw5VO9Yn4HqFQDqZMjVBp496CEY+aDGrwuj7dY/FjPR97A== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4hcKkC11KzzVXm for ; Sat, 05 Sep 2026 04:15:15 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id 3e987 by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Sat, 05 Sep 2026 04:15:15 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Justin Hibbits Subject: git: 564fe16c972d - main - powerpc/booke: Use a dedicated critical exception stack List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: jhibbits X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 564fe16c972dc039fc7b01b38dd913fe13eb1278 Auto-Submitted: auto-generated Date: Sat, 05 Sep 2026 04:15:15 +0000 Message-Id: <6a9b9753.3e987.76a5de6f@gitrepo.freebsd.org> The branch main has been updated by jhibbits: URL: https://cgit.FreeBSD.org/src/commit/?id=564fe16c972dc039fc7b01b38dd913fe13eb1278 commit 564fe16c972dc039fc7b01b38dd913fe13eb1278 Author: Justin Hibbits AuthorDate: 2026-09-04 02:48:35 +0000 Commit: Justin Hibbits CommitDate: 2026-09-05 04:09:22 +0000 powerpc/booke: Use a dedicated critical exception stack A critical exception, such as a watchdog, can trigger at any time, including the middle of a standard exception prologue or epilogue, so GPRs, including %r1 (the stack pointer) cannot be trusted at all. Instead, use a private stack pointer for critical interrupts. Each CPU now has its own critical exception stack, with the boot stack in the bss. --- sys/powerpc/booke/booke_machdep.c | 18 ++++++++++++++ sys/powerpc/booke/trap_subr.S | 50 ++++++++++++++++++++++++++++++--------- sys/powerpc/include/pcpu.h | 12 ++++++++-- sys/powerpc/powerpc/genassym.c | 1 + 4 files changed, 68 insertions(+), 13 deletions(-) diff --git a/sys/powerpc/booke/booke_machdep.c b/sys/powerpc/booke/booke_machdep.c index 467eb2ab9638..99b14bc9e974 100644 --- a/sys/powerpc/booke/booke_machdep.c +++ b/sys/powerpc/booke/booke_machdep.c @@ -97,6 +97,7 @@ #include #include #include +#include #include #include #include @@ -390,13 +391,30 @@ booke_init(u_long arg1, u_long arg2) #define RES_GRANULE cacheline_size extern uintptr_t tlb0_miss_locks[]; +/* + * Stack for critical-class interrupts taken in kernel mode on the boot CPU. + * It is static because the boot CPU is initialised long before the VM is + * running; APs reach cpu_pcpu_init() at SI_SUB_CPU and can allocate. Sizing a + * MAXCPU array here would reserve megabytes for CPUs that do not exist. + */ +static char booke_boot_critstack[BOOKE_CRITSTACK_SIZE] __aligned(16); +static bool booke_boot_critstack_used; + /* Initialise a struct pcpu. */ void cpu_pcpu_init(struct pcpu *pcpu, int cpuid, size_t sz) { + char *critstack; pcpu->pc_booke.tid_next = TID_MIN; + if (!booke_boot_critstack_used) { + booke_boot_critstack_used = true; + critstack = booke_boot_critstack; + } else + critstack = malloc(BOOKE_CRITSTACK_SIZE, M_DEVBUF, M_WAITOK); + pcpu->pc_booke.critstack = critstack + BOOKE_CRITSTACK_SIZE; + #ifdef SMP uintptr_t *ptr; int words_per_gran = RES_GRANULE / sizeof(uintptr_t); diff --git a/sys/powerpc/booke/trap_subr.S b/sys/powerpc/booke/trap_subr.S index 4bfd2118fb8f..10babefbb2f8 100644 --- a/sys/powerpc/booke/trap_subr.S +++ b/sys/powerpc/booke/trap_subr.S @@ -145,23 +145,44 @@ mfspr %r31, SPR_ESR; \ STORE %r30, (savearea+CPUSAVE_BOOKE_DEAR)(%r1); \ STORE %r31, (savearea+CPUSAVE_BOOKE_ESR)(%r1); \ + mfspr %r30, SPR_SRR0; \ + mfspr %r31, SPR_SRR1; \ + STORE %r30, (savearea+BOOKE_CRITSAVE_SRR0)(%r1); \ + STORE %r31, (savearea+BOOKE_CRITSAVE_SRR1)(%r1); \ mfspr %r30, isrr0; \ mfspr %r31, isrr1; /* MSR at interrupt time */ \ STORE %r30, (savearea+CPUSAVE_SRR0)(%r1); \ STORE %r31, (savearea+CPUSAVE_SRR1)(%r1); \ - mfspr %r30, SPR_SRR0; \ - mfspr %r31, SPR_SRR1; /* MSR at interrupt time */ \ - STORE %r30, (savearea+BOOKE_CRITSAVE_SRR0)(%r1); \ - STORE %r31, (savearea+BOOKE_CRITSAVE_SRR1)(%r1); \ isync; \ - mfspr %r1, sprg_sp; /* Restore SP */ \ mfcr %r30; /* Save CR */ \ - /* switch to per-thread kstack if intr taken in user mode */ \ mtcr %r31; /* MSR at interrupt time */ \ - bf 17, 1f; \ + /* \ + * Never inherit the interrupted r1. A critical interrupt can \ + * land partway through a non-critical prolog, where r1 holds \ + * the pcpu pointer rather than a stack, and laying a frame \ + * there walks straight into the adjacent CPU's pcpu. \ + */ \ GET_CPUINFO(%r1); /* Per-cpu structure */ \ + bf 17, 1f; \ LOAD %r1, PC_CURPCB(%r1); /* Per-thread kernel stack */ \ -1: + b 2f; \ +1: LOAD %r1, PC_BOOKE_CRITSTACK(%r1); \ +2: + +/* + * Put back the SRR0-1 saved by STANDARD_CRIT_PROLOG. + * + * A critical interrupt can land in the middle of a non-critical prolog, + * before it has stashed SRR0-1, or between the mtsrr0/mtsrr1 and the rfi of + * a TLB miss return. Nested exceptions taken by the C dispatcher clobber + * them either way, so restore before returning. + */ +#define CRIT_SRR_RESTORE \ + GET_CPUINFO(%r3); \ + LOAD %r4, (PC_BOOKE_CRITSAVE+BOOKE_CRITSAVE_SRR0)(%r3); \ + LOAD %r5, (PC_BOOKE_CRITSAVE+BOOKE_CRITSAVE_SRR1)(%r3); \ + mtspr SPR_SRR0, %r4; \ + mtspr SPR_SRR1, %r5 /* * FRAME_SETUP assumes: @@ -519,6 +540,7 @@ INTERRUPT(int_critical_input) addi %r3, %r1, CALLSIZE bl CNAME(powerpc_interrupt) TOC_RESTORE + CRIT_SRR_RESTORE FRAME_LEAVE(SPR_CSRR0, SPR_CSRR1) rfci @@ -613,9 +635,15 @@ INTERRUPT(int_fixed_interval_timer) * Watchdog interrupt ****************************************************************************/ INTERRUPT(int_watchdog) - STANDARD_CRIT_PROLOG(SPR_SPRG1, PC_BOOKE_CRITSAVE, SPR_CSRR0, SPR_CSRR1) - FRAME_SETUP(SPR_SPRG1, PC_BOOKE_CRITSAVE, EXC_WDOG) - b trap_common + STANDARD_CRIT_PROLOG(SPR_SPRG2, PC_BOOKE_CRITSAVE, SPR_CSRR0, SPR_CSRR1) + FRAME_SETUP(SPR_SPRG2, PC_BOOKE_CRITSAVE, EXC_WDOG) + GET_TOCBASE(%r2) + addi %r3, %r1, CALLSIZE + bl CNAME(powerpc_interrupt) + TOC_RESTORE + CRIT_SRR_RESTORE + FRAME_LEAVE(SPR_CSRR0, SPR_CSRR1) + rfci /***************************************************************************** diff --git a/sys/powerpc/include/pcpu.h b/sys/powerpc/include/pcpu.h index f2a9bd0080b5..18f0e3209b92 100644 --- a/sys/powerpc/include/pcpu.h +++ b/sys/powerpc/include/pcpu.h @@ -85,10 +85,17 @@ struct pvo_entry; #define BOOKE_TLB_SAVELEN 16 #define BOOKE_TLBSAVE_LEN (BOOKE_TLB_SAVELEN * BOOKE_TLB_MAXNEST) +/* + * Stack used by critical-class interrupts taken in kernel mode. Sized for the + * full panic path (powerpc_interrupt -> trap_fatal -> printf -> backtrace), + * since that is the only thing that ever runs on it. + */ +#define BOOKE_CRITSTACK_SIZE 16384 + #ifdef __powerpc64__ -#define BOOKE_PCPU_PAD 901 +#define BOOKE_PCPU_PAD 893 #else -#define BOOKE_PCPU_PAD 365 +#define BOOKE_PCPU_PAD 361 #endif #define PCPU_MD_BOOKE_FIELDS \ register_t critsave[BOOKE_CRITSAVE_LEN]; \ @@ -96,6 +103,7 @@ struct pvo_entry; register_t tlbsave[BOOKE_TLBSAVE_LEN]; \ register_t tlb_level; \ uintptr_t *tlb_lock; \ + void *critstack; \ int tid_next; \ char __pad[BOOKE_PCPU_PAD]; diff --git a/sys/powerpc/powerpc/genassym.c b/sys/powerpc/powerpc/genassym.c index bc39e086bfa0..e8c08867d778 100644 --- a/sys/powerpc/powerpc/genassym.c +++ b/sys/powerpc/powerpc/genassym.c @@ -69,6 +69,7 @@ ASSYM(PC_BOOKE_MCHKSAVE, offsetof(struct pcpu, pc_booke.mchksave)); ASSYM(PC_BOOKE_TLBSAVE, offsetof(struct pcpu, pc_booke.tlbsave)); ASSYM(PC_BOOKE_TLB_LEVEL, offsetof(struct pcpu, pc_booke.tlb_level)); ASSYM(PC_BOOKE_TLB_LOCK, offsetof(struct pcpu, pc_booke.tlb_lock)); +ASSYM(PC_BOOKE_CRITSTACK, offsetof(struct pcpu, pc_booke.critstack)); #endif ASSYM(CPUSAVE_R27, CPUSAVE_R27*sizeof(register_t));