From nobody Wed Sep 14 15:57:43 2022 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MSQ2w3Xp3z4cRWl; Wed, 14 Sep 2022 15:57:44 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4MSQ2w1CBsz3GNS; Wed, 14 Sep 2022 15:57:44 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1663171064; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=ktlvIF8Ts6o/cEwKrbHLhTcwnedJdhqcJwrS6Z82qnI=; b=jTRGaoKHeahOLXJ4jvZyo8MrwPZrGCwt7xmg112TPVnfWpffYP3bjfJcWnBOPvxLafUMa7 D5LfwJZTzsJPeT9bId1GPBEsWJihBM42uvrGQHEB+tsphNNgyJIJsbE9inerlaFbNLyvGU pegrJYsmnzyrX7PccK3QKDO51jycZMiCr90Ea2f3dskaHRyoMMyLaH7ja31o6xS1kT8DAw 9quKPAvazxe0W0WQZzl4w2C0w1sTHI6XJEqRx4XXkoXoJ0+eVoh2gVDG95RtfGIwKMowHE 91YCqdzQ2HfndXIRR3jxT2tyIgVW3c3BA9+h8niQMzDm3Bx7/G5N1KWO9d92IA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4MSQ2v60HVzk2M; Wed, 14 Sep 2022 15:57:43 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 28EFvhGh010651; Wed, 14 Sep 2022 15:57:43 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 28EFvhQw010650; Wed, 14 Sep 2022 15:57:43 GMT (envelope-from git) Date: Wed, 14 Sep 2022 15:57:43 GMT Message-Id: <202209141557.28EFvhQw010650@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Konstantin Belousov Subject: git: 49587b8fb4d1 - main - i386 copyout/in_fast: handle page fault from KVA access List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: kib X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 49587b8fb4d12166d30efb7111d32f4606a1e73b Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1663171064; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=ktlvIF8Ts6o/cEwKrbHLhTcwnedJdhqcJwrS6Z82qnI=; b=kMsKiz8Ws+iq4XdL7Cgfwl+uBdjZPJ/UQaM2AppMpgwAiv9JOVe5gQs89IjsOYXbRbOVCU Klz4aaAwyLBEU8H8Wl1H/01BT6pwBBgbfaDNf7sqjVALuG4fOZz1oizPXQ+z88YWYyAwvh L3N72DKwHI4GaK356ef9+V7mjW9uAgm8S+pvRoAhSAL3qsH37H0oVjAWVCtEiUigAPI5YA 5C5yTsn1yofcM1oCqMCc1O12nd5K3dRy5Fa8ttVtlQyelKGzz7F+vmotm8Mg/5BCNjQrpf Fcb3Qx/WuqgKI/HkJfElSv6Y7P4jT5xFRA0zW433EggVXv6w+kHKeuLUF/3FqQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1663171064; a=rsa-sha256; cv=none; b=EK0s3UUNQb498l4J1JAOFwU1hZsrX4V5mSzLFLraYMFFaBC1aBbbQ3xIK79fzqcOkokNVh C1g4LrXj4iWcWpaAZPCHk4JTDnudZ6k9B6uW6S0OEGFoUdU7xqRIPPouvQmMpFq4+/pugB A4QvvCdr7J0JZHTifKfoguclzCBqpKIuP9HTUdR0S0a/VruHj/qxfuJ8GbZdwRIIkXYsZ0 dS/0NNSYYV8n1oLv9KOFDXSlPXDsM1eckwJQkm5eyE4/I05A4xzM4I5XY7dFvupA81CFb+ GhS/5/b3CHZSwHqWRCbRBqKflopMFf8PziP1fWTdvbFngUHZT5I2fdTg44Aimw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by kib: URL: https://cgit.FreeBSD.org/src/commit/?id=49587b8fb4d12166d30efb7111d32f4606a1e73b commit 49587b8fb4d12166d30efb7111d32f4606a1e73b Author: Konstantin Belousov AuthorDate: 2022-09-08 20:15:32 +0000 Commit: Konstantin Belousov CommitDate: 2022-09-14 15:46:32 +0000 i386 copyout/in_fast: handle page fault from KVA access by delegating the work to the slow path. Some kernel memory, like pipe buffers, is pageable. We must not enable interrupts, and consequently, preemption, while in critical section in the fast copyout path, because we use pcpu buffers. If page fault occurs while copying from the pcpu copyout_buf to kernel memory, abort fast path and delegate work to the slow implementation. In collaboration with: pho, tijl Sponsored by: The FreeBSD Foundation MFC after: 1 week --- sys/i386/i386/copyout_fast.s | 21 +++++++++------------ sys/i386/i386/exception.s | 25 +++++++++++++++---------- 2 files changed, 24 insertions(+), 22 deletions(-) diff --git a/sys/i386/i386/copyout_fast.s b/sys/i386/i386/copyout_fast.s index 70647fe7613d..4391f36b18e4 100644 --- a/sys/i386/i386/copyout_fast.s +++ b/sys/i386/i386/copyout_fast.s @@ -47,32 +47,30 @@ ENTRY(copyout_fast) pushl %ebx movl 20(%ebp),%ebx /* KCR3 */ - movl PCPU(CURPCB),%edx - movl PCB_CR3(%edx),%edx /* UCR3 */ /* bcopy(%esi = kaddr, %edi = PCPU(copyout_buf), %ecx = len) */ movl 16(%ebp),%ecx movl 8(%ebp),%esi + movl %esp,%eax + movl $copyout_fault,%edx + cli movl PCPU(COPYOUT_BUF),%edi - rep; movsb +pf_y1: rep; movsb movl 16(%ebp),%ecx /* len */ movl PCPU(COPYOUT_BUF),%esi /* kaddr */ movl 12(%ebp),%edi /* uaddr */ - - movl %esp,%eax movl PCPU(TRAMPSTK),%esp - + movl PCPU(CURPCB),%edx + movl PCB_CR3(%edx),%edx /* UCR3 */ movl %edx,%cr3 movl $copyout_fault,%edx - /* bcopy(%esi = PCPU(copyout_buf), %edi = udaddr, %ecx = len) */ pf_x1: rep; movsb movl %ebx,%cr3 movl %eax,%esp sti - xorl %eax,%eax popl %ebx popl %edi @@ -93,10 +91,10 @@ ENTRY(copyin_fast) movl PCB_CR3(%eax),%edx /* UCR3 */ movl 16(%ebp),%ecx /* len */ movl 8(%ebp),%esi /* udaddr */ + movl %esp,%eax + cli movl PCPU(COPYOUT_BUF),%edi /* kaddr */ - - movl %esp,%eax movl PCPU(TRAMPSTK),%esp movl %edx,%cr3 movl $copyout_fault,%edx @@ -110,10 +108,9 @@ pf_x2: rep; movsb movl 16(%ebp),%ecx movl 12(%ebp),%edi movl PCPU(COPYOUT_BUF),%esi - rep; movsb +pf_y2: rep; movsb sti - xorl %eax,%eax popl %ebx popl %edi diff --git a/sys/i386/i386/exception.s b/sys/i386/i386/exception.s index f8d61c884a21..e83d95e6afe1 100644 --- a/sys/i386/i386/exception.s +++ b/sys/i386/i386/exception.s @@ -130,15 +130,15 @@ IDTVEC(prot) jmp irettraps IDTVEC(page) testl $PSL_VM, TF_EFLAGS-TF_ERR(%esp) - jnz 4f + jnz 5f testb $SEL_RPL_MASK, TF_CS-TF_ERR(%esp) - jnz 4f + jnz 5f cmpl $PMAP_TRM_MIN_ADDRESS, TF_EIP-TF_ERR(%esp) - jb 4f + jb 5f pushl %eax movl TF_EIP-TF_ERR+4(%esp), %eax addl $1f, %eax - call 5f + call 6f 1: cmpl $pf_x1, %eax je 2f cmpl $pf_x2, %eax @@ -154,16 +154,21 @@ IDTVEC(page) cmpl $pf_x7, %eax je 2f cmpl $pf_x8, %eax - jne 3f -2: popl %eax - movl %ebx, %cr3 + je 2f + cmpl $pf_y1, %eax + je 3f + cmpl $pf_y2, %eax + je 3f + jmp 4f +2: movl %ebx, %cr3 +3: popl %eax movl %edx, TF_EIP-TF_ERR(%esp) addl $4, %esp iret -3: popl %eax -4: pushl $T_PAGEFLT +4: popl %eax +5: pushl $T_PAGEFLT jmp alltraps -5: subl (%esp), %eax +6: subl (%esp), %eax retl IDTVEC(rsvd_pti) IDTVEC(rsvd)