From nobody Thu Apr 10 12:47:20 2025 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZYKMr56TLz5svn5; Thu, 10 Apr 2025 12:47:20 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZYKMr2Nx7z3ntn; Thu, 10 Apr 2025 12:47:20 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1744289240; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=t3FTZb17QfdUkGNmQ4TnEDOq6tldCyqLfJd2lwsSLCs=; b=Lmd7QyLfr6cLFz4yVJ7NwOrbPy6svH9f+YQofA0xy2T4sodPzz3WxzeGV/WHCVGyhpdcff kiElx11DtTFV2CyMt8628f+00g8uq3TdUhPpfenmajbMTaLniOYKbT3leWa0OXVbncNouA 0z2Sn49ROERE81zQTOs0t4/d70PIHNw7IpV85g462Hl8GUrjX/NcllCkpHvL9F4npoQDRT ESPnwVzaJIkKbLz+PACYZKntd97cpPxz90iHirMgErXK6N0jeWK9GFiM+fJbshNExAy/dF 1AzVu8tct+AAd4lCPe12sVbjvCxhxyrD5MV1lq4W2dwoG0QETzk8KVx0fYDegA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1744289240; a=rsa-sha256; cv=none; b=KXniPgKDww/XPV+pmjpTdEZSKsJZHbLT0GDEJ+FttYEKRhB3hgje0Dsfp73skDuZCnmBZC GnoulCQD90aTI5BH54nuu5czl3G7Vglul6mqU8nap70KNCU6cU32uUEZ83yKB64Sfs1C2/ i/Fnt7duK7O7KyGkWh1vLxtt60TfjZLSkBv6R633KgyKMN5uBaK3g5QwotE8qXHsK774fm ht9ZLau1ooeFtuTzKl8nYgU/tN8dFBMDuzOUvI1CyxB+1P1/fYoZX/XQ8pk4GW9ogaL9y0 hV+1yp6KclQEdozUkmoV+2SusLFYK5xSelyTshayzeYIup3EEvF25Cm+uuOgQw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1744289240; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=t3FTZb17QfdUkGNmQ4TnEDOq6tldCyqLfJd2lwsSLCs=; b=QT15isTyNgEKIJPyVjqLi4AilwIXqLUwWT37N6+4oZo9O2k8zJp1WXCtcSFbqeK5e0euj6 YOphraFdbzc2YdS+kkpaG4zWAwEtjXUQ1RhXea9+G6JH49cmo/VX/H8b9TDGfJh4busrWq YA9c3vzf3Zj+NDMv2FHkgkcenC+1OpKkAUVoMVSMPvJGxXbza65in7NZQq7HVV4vbQfeaK EQJKAwG87n3/KfpmnQaR1c0BG/dCT7xDFDKyTzBrF9X/aFGscHleFQHagVFku3Lh87fkcO sgXRLdmXOP/+ijpRn9/OaCtUJWTCNVcOZHNVOUL6ekyW6XbCTKjkotovQN12pw== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZYKMr14Wpz13Bj; Thu, 10 Apr 2025 12:47:20 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 53AClKGw015032; Thu, 10 Apr 2025 12:47:20 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 53AClKUm015029; Thu, 10 Apr 2025 12:47:20 GMT (envelope-from git) Date: Thu, 10 Apr 2025 12:47:20 GMT Message-Id: <202504101247.53AClKUm015029@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Mark Johnston Subject: git: ae10431c9833 - main - vm_page: Allow PG_NOFREE pages to be freed List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: markj X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: ae10431c9833bd6b176afe4d8021d233fd985107 Auto-Submitted: auto-generated The branch main has been updated by markj: URL: https://cgit.FreeBSD.org/src/commit/?id=ae10431c9833bd6b176afe4d8021d233fd985107 commit ae10431c9833bd6b176afe4d8021d233fd985107 Author: Mark Johnston AuthorDate: 2025-04-10 12:43:12 +0000 Commit: Mark Johnston CommitDate: 2025-04-10 12:47:05 +0000 vm_page: Allow PG_NOFREE pages to be freed There is at least one case where we need to support it: kmem_malloc() might need to allocate multiple pages to satisfy a NOFREE allocation, which it implements by calling vm_page_alloc() in a loop. If it fails part-way though, it needs to free already-allocated pages, but this was illegal. Convert the bump allocator to a linked list; (ab)use the pindex field of each page in the list to store the number of contiguous pages in the block. (Originally I added a new plinks member for this purpose, but it's not safe to use that until after vm_page_dequeue() is called due to lazy page queue removal.) Then, modify vm_page_free() to support freeing pages to this list. While here, add a __noinline qualifier to vm_page_alloc_nofree_domain() to ensure that it doesn't get inlined into a hot path. Reported by: syzbot+93bc9edd2d0f22ae426a@syzkaller.appspotmail.com Reviewed by: bnovkov, kib Fixes: a8693e89e3e4 ("vm: Introduce vm_page_alloc_nofree_domain") Differential Revision: https://reviews.freebsd.org/D49480 --- sys/vm/vm_page.c | 56 ++++++++++++++++++++++++++++++++++++++------------- sys/vm/vm_pagequeue.h | 5 +---- 2 files changed, 43 insertions(+), 18 deletions(-) diff --git a/sys/vm/vm_page.c b/sys/vm/vm_page.c index 741c45490d96..5214b3c956ba 100644 --- a/sys/vm/vm_page.c +++ b/sys/vm/vm_page.c @@ -2648,40 +2648,66 @@ found: * the routine will try to fetch a new one from the freelists * and discard the old one. */ -static vm_page_t +static vm_page_t __noinline vm_page_alloc_nofree_domain(int domain, int req) { vm_page_t m; struct vm_domain *vmd; - struct vm_nofreeq *nqp; KASSERT((req & VM_ALLOC_NOFREE) != 0, ("invalid request %#x", req)); vmd = VM_DOMAIN(domain); - nqp = &vmd->vmd_nofreeq; vm_domain_free_lock(vmd); - if (nqp->offs >= (1 << VM_NOFREE_IMPORT_ORDER) || nqp->ma == NULL) { - if (!vm_domain_allocate(vmd, req, - 1 << VM_NOFREE_IMPORT_ORDER)) { + if (TAILQ_EMPTY(&vmd->vmd_nofreeq)) { + int count; + + count = 1 << VM_NOFREE_IMPORT_ORDER; + if (!vm_domain_allocate(vmd, req, count)) { vm_domain_free_unlock(vmd); return (NULL); } - nqp->ma = vm_phys_alloc_pages(domain, VM_FREEPOOL_DEFAULT, + m = vm_phys_alloc_pages(domain, VM_FREEPOOL_DEFAULT, VM_NOFREE_IMPORT_ORDER); - if (nqp->ma == NULL) { - vm_domain_freecnt_inc(vmd, 1 << VM_NOFREE_IMPORT_ORDER); + if (m == NULL) { + vm_domain_freecnt_inc(vmd, count); vm_domain_free_unlock(vmd); return (NULL); } - nqp->offs = 0; + m->pindex = count; + TAILQ_INSERT_HEAD(&vmd->vmd_nofreeq, m, listq); + VM_CNT_ADD(v_nofree_count, count); + } + m = TAILQ_FIRST(&vmd->vmd_nofreeq); + TAILQ_REMOVE(&vmd->vmd_nofreeq, m, listq); + if (m->pindex > 1) { + vm_page_t m_next; + + m_next = &m[1]; + m_next->pindex = m->pindex - 1; + TAILQ_INSERT_HEAD(&vmd->vmd_nofreeq, m_next, listq); } - m = &nqp->ma[nqp->offs++]; vm_domain_free_unlock(vmd); - VM_CNT_ADD(v_nofree_count, 1); + VM_CNT_ADD(v_nofree_count, -1); return (m); } +/* + * Though a NOFREE page by definition should not be freed, we support putting + * them aside for future NOFREE allocations. This enables code which allocates + * NOFREE pages for some purpose but then encounters an error and releases + * resources. + */ +static void __noinline +vm_page_free_nofree(struct vm_domain *vmd, vm_page_t m) +{ + vm_domain_free_lock(vmd); + m->pindex = 1; + TAILQ_INSERT_HEAD(&vmd->vmd_nofreeq, m, listq); + vm_domain_free_unlock(vmd); + VM_CNT_ADD(v_nofree_count, 1); +} + vm_page_t vm_page_alloc_noobj(int req) { @@ -4145,8 +4171,6 @@ vm_page_free_prep(vm_page_t m) m, i, (uintmax_t)*p)); } #endif - KASSERT((m->flags & PG_NOFREE) == 0, - ("%s: attempting to free a PG_NOFREE page", __func__)); if ((m->oflags & VPO_UNMANAGED) == 0) { KASSERT(!pmap_page_is_mapped(m), ("vm_page_free_prep: freeing mapped page %p", m)); @@ -4230,6 +4254,10 @@ vm_page_free_toq(vm_page_t m) return; vmd = vm_pagequeue_domain(m); + if (__predict_false((m->flags & PG_NOFREE) != 0)) { + vm_page_free_nofree(vmd, m); + return; + } zone = vmd->vmd_pgcache[m->pool].zone; if ((m->flags & PG_PCPU_CACHE) != 0 && zone != NULL) { uma_zfree(zone, m); diff --git a/sys/vm/vm_pagequeue.h b/sys/vm/vm_pagequeue.h index 72fd1bb47318..cbbd27389662 100644 --- a/sys/vm/vm_pagequeue.h +++ b/sys/vm/vm_pagequeue.h @@ -247,10 +247,7 @@ struct vm_domain { u_int vmd_domain; /* (c) Domain number. */ u_int vmd_page_count; /* (c) Total page count. */ long vmd_segs; /* (c) bitmask of the segments */ - struct vm_nofreeq { - vm_page_t ma; - int offs; - } vmd_nofreeq; /* (f) NOFREE page bump allocator. */ + struct pglist vmd_nofreeq; /* (f) NOFREE page bump allocator. */ u_int __aligned(CACHE_LINE_SIZE) vmd_free_count; /* (a,f) free page count */ u_int vmd_pageout_deficit; /* (a) Estimated number of pages deficit */ uint8_t vmd_pad[CACHE_LINE_SIZE - (sizeof(u_int) * 2)];