From nobody Tue Apr 01 06:10:55 2025 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZRd0b4nP7z5s1Wj; Tue, 01 Apr 2025 06:10:55 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZRd0b43XJz3Q6Q; Tue, 01 Apr 2025 06:10:55 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743487855; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=KKtCMeQHB22jBKSCxhFyQAxCAq7PaCTeCi5yiIJf0K4=; b=QhTiBBnsgrxDcwDvNoPItzR1cumVpazV8hCicBd60sAcMlfNnLQrtSkJCVldFltpZNpLvq 3UOCdLWdkqn1HLQj6FE8jAO2WotZhTr5Ns5gK9q8IeeeA6xbi2IYEh1s7csrl4cVFu3dM+ DJvrdf+UQa4HBrc+PITr2x81Uoasmor/K40+jEeUrcITG9t3G96rnkSYnZMepXQcyu3AMT cE28m2xF3lAZEUp3aSaFP4mznXunzwCzZ590v2/9aCfwJvt+x8Y9UTx1UDLbGyQKXP/VOf 8yi0BG9jkdFFKcy36xeKY9ZcLEO8TjIPJ/4MT+qH1v9V7SgzOPWut8tqf8QIjA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1743487855; a=rsa-sha256; cv=none; b=wWFTniV/8NGhnlkA1bUhjzznEJhJzuXK1UkSOxQ1OBQ4bU/5qK+D8Oi57uRGQvYZ2BOQaS GFdRKhM1ukqKr2/Hu8l7k9V/73FHdIpMWBD3Eib1p6ddjg2GFf5zDUd61HNMV31xEPLnaw y18QDsuqiCDufTine5W8oUUJarKe8lbQZd9u3qh2XxK2cOreNX9ay0irzseiQPeJrTlJj3 6NWJCj8KaxRL58XU5cLnoCzeVvE26eO8xDBMOTpygqsKq4WzLFPf5t/PoKjsZBYM3oac/f o6OjeKz9UpWhR/dOViYHV+707Q4kAqhwBRno+okjMgGPJjDMYkofH2KRvQqo8g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1743487855; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=KKtCMeQHB22jBKSCxhFyQAxCAq7PaCTeCi5yiIJf0K4=; b=LG7Kn90f0dPeQECIqtehpPIXfpGS9IxdVjVh2o17E5LZqOZuBANVUMoSyReJl6pjz0EBBt vDuk3AmBrCDQJEdbGqmF1WZXqfiYHaxFdIH8iiuuubOSCrWgF6Gl4Q+RI0m5E4ZbGmS3nv hfzHysaw3WbTvTWi+0E5F/jrEl4WWS43KjT0Jeu1S4IDjB6P6RrSTCzYIN+VcLcH9iXqfy auUw3+XSptt83w5SsBVHVnuKzvloQyffpbHvE9EbG/RuIALAFDLLE29tPu3i1nPel7mv9G byy12acrCUdGGBZbjxiPTB9RUTjSuRkTPyGJHSu6w2FnyItce/YCJadtmXwNIA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZRd0b3V14z3v2; Tue, 01 Apr 2025 06:10:55 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 5316At1I055451; Tue, 1 Apr 2025 06:10:55 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 5316AtcP055448; Tue, 1 Apr 2025 06:10:55 GMT (envelope-from git) Date: Tue, 1 Apr 2025 06:10:55 GMT Message-Id: <202504010610.5316AtcP055448@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Doug Moore Subject: git: 7becd87c988c - main - vm_grab: use iterator for grab lookup List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: dougm X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 7becd87c988c090bdc41a177bf06ad2894f79a5b Auto-Submitted: auto-generated The branch main has been updated by dougm: URL: https://cgit.FreeBSD.org/src/commit/?id=7becd87c988c090bdc41a177bf06ad2894f79a5b commit 7becd87c988c090bdc41a177bf06ad2894f79a5b Author: Doug Moore AuthorDate: 2025-04-01 06:09:34 +0000 Commit: Doug Moore CommitDate: 2025-04-01 06:09:34 +0000 vm_grab: use iterator for grab lookup When a page lookup fails in a vm_page_grab operation, use the results of the failed search, stored in an iterator, to begin the search for a predecessor to use in a call to vm_page_alloc_after(), to avoid doing that search from the root in vm_page_alloc(), as happens now. Reviewed by: kib Differential Revision: https://reviews.freebsd.org/D49371 --- sys/vm/vm_page.c | 145 ++++++++++++++++++++++++++++++++++--------------------- 1 file changed, 89 insertions(+), 56 deletions(-) diff --git a/sys/vm/vm_page.c b/sys/vm/vm_page.c index 4d9a57544487..f351f60f833c 100644 --- a/sys/vm/vm_page.c +++ b/sys/vm/vm_page.c @@ -4823,46 +4823,70 @@ vm_page_grab_pflags(int allocflags) } /* - * Grab a page, waiting until we are waken up due to the page - * changing state. We keep on waiting, if the page continues - * to be in the object. If the page doesn't exist, first allocate it - * and then conditionally zero it. + * Grab a page, waiting until we are woken up due to the page changing state. + * We keep on waiting, if the page continues to be in the object, unless + * allocflags forbid waiting. * - * This routine may sleep. + * The object must be locked on entry. This routine may sleep. The lock will, + * however, be released and reacquired if the routine sleeps. * - * The object must be locked on entry. The lock will, however, be released - * and reacquired if the routine sleeps. + * Return a grabbed page, or NULL. Set *found if a page was found, whether or + * not it was grabbed. + */ +static inline vm_page_t +vm_page_grab_lookup(struct pctrie_iter *pages, vm_object_t object, + vm_pindex_t pindex, int allocflags, bool *found) +{ + vm_page_t m; + + while ((*found = (m = vm_radix_iter_lookup(pages, pindex)) != NULL) && + !vm_page_tryacquire(m, allocflags)) { + if (!vm_page_grab_sleep(object, m, pindex, "pgrbwt", + allocflags, true)) + return (NULL); + pctrie_iter_reset(pages); + } + return (m); +} + +/* + * Grab a page. Keep on waiting, as long as the page exists in the object. If + * the page doesn't exist, first allocate it and then conditionally zero it. + * + * The object must be locked on entry. This routine may sleep. The lock will, + * however, be released and reacquired if the routine sleeps. */ vm_page_t vm_page_grab(vm_object_t object, vm_pindex_t pindex, int allocflags) { - vm_page_t m; + struct pctrie_iter pages; + vm_page_t m, mpred; + bool found; VM_OBJECT_ASSERT_WLOCKED(object); vm_page_grab_check(allocflags); -retrylookup: - if ((m = vm_page_lookup(object, pindex)) != NULL) { - if (!vm_page_tryacquire(m, allocflags)) { - if (vm_page_grab_sleep(object, m, pindex, "pgrbwt", - allocflags, true)) - goto retrylookup; + vm_page_iter_init(&pages, object); + while ((m = vm_page_grab_lookup( + &pages, object, pindex, allocflags, &found)) == NULL) { + if ((allocflags & VM_ALLOC_NOCREAT) != 0) + return (NULL); + if (found && + (allocflags & (VM_ALLOC_NOWAIT | VM_ALLOC_WAITFAIL)) != 0) return (NULL); + mpred = vm_radix_iter_lookup_le(&pages, pindex); + m = vm_page_alloc_after(object, pindex, + vm_page_grab_pflags(allocflags), mpred); + if (m != NULL) { + if ((allocflags & VM_ALLOC_ZERO) != 0 && + (m->flags & PG_ZERO) == 0) + pmap_zero_page(m); + break; } - goto out; - } - if ((allocflags & VM_ALLOC_NOCREAT) != 0) - return (NULL); - m = vm_page_alloc(object, pindex, vm_page_grab_pflags(allocflags)); - if (m == NULL) { - if ((allocflags & (VM_ALLOC_NOWAIT | VM_ALLOC_WAITFAIL)) != 0) + if ((allocflags & + (VM_ALLOC_NOWAIT | VM_ALLOC_WAITFAIL)) != 0) return (NULL); - goto retrylookup; } - if (allocflags & VM_ALLOC_ZERO && (m->flags & PG_ZERO) == 0) - pmap_zero_page(m); - -out: vm_page_grab_release(m, allocflags); return (m); @@ -5054,48 +5078,57 @@ out: } /* - * Fill a partial page with zeroes. The object write lock is held on entry and - * exit, but may be temporarily released. + * Grab a page. Keep on waiting, as long as the page exists in the object. If + * the page doesn't exist, and the pager has it, allocate it and zero part of + * it. + * + * The object must be locked on entry. This routine may sleep. The lock will, + * however, be released and reacquired if the routine sleeps. */ int vm_page_grab_zero_partial(vm_object_t object, vm_pindex_t pindex, int base, int end) { - vm_page_t m; - int rv; + struct pctrie_iter pages; + vm_page_t m, mpred; + int allocflags, rv; + bool found; VM_OBJECT_ASSERT_WLOCKED(object); KASSERT(base >= 0, ("%s: base %d", __func__, base)); KASSERT(end - base <= PAGE_SIZE, ("%s: base %d end %d", __func__, base, end)); -retry: - m = vm_page_grab(object, pindex, VM_ALLOC_NOCREAT); - if (m != NULL) { - MPASS(vm_page_all_valid(m)); - } else if (vm_pager_has_page(object, pindex, NULL, NULL)) { - m = vm_page_alloc(object, pindex, - VM_ALLOC_NORMAL | VM_ALLOC_WAITFAIL); - if (m == NULL) - goto retry; - vm_object_pip_add(object, 1); - VM_OBJECT_WUNLOCK(object); - rv = vm_pager_get_pages(object, &m, 1, NULL, NULL); - VM_OBJECT_WLOCK(object); - vm_object_pip_wakeup(object); - if (rv != VM_PAGER_OK) { - vm_page_free(m); - return (EIO); - } + allocflags = VM_ALLOC_NOCREAT | VM_ALLOC_NORMAL | VM_ALLOC_WAITFAIL; + vm_page_iter_init(&pages, object); + while ((m = vm_page_grab_lookup( + &pages, object, pindex, allocflags, &found)) == NULL) { + if (!vm_pager_has_page(object, pindex, NULL, NULL)) + return (0); + mpred = vm_radix_iter_lookup_le(&pages, pindex); + m = vm_page_alloc_after(object, pindex, + vm_page_grab_pflags(allocflags), mpred); + if (m != NULL) { + vm_object_pip_add(object, 1); + VM_OBJECT_WUNLOCK(object); + rv = vm_pager_get_pages(object, &m, 1, NULL, NULL); + VM_OBJECT_WLOCK(object); + vm_object_pip_wakeup(object); + if (rv != VM_PAGER_OK) { + vm_page_free(m); + return (EIO); + } - /* - * Since the page was not resident, and therefore not recently - * accessed, immediately enqueue it for asynchronous laundering. - * The current operation is not regarded as an access. - */ - vm_page_launder(m); - } else - return (0); + /* + * Since the page was not resident, and therefore not + * recently accessed, immediately enqueue it for + * asynchronous laundering. The current operation is + * not regarded as an access. + */ + vm_page_launder(m); + break; + } + } pmap_zero_page_area(m, base, end - base); KASSERT(vm_page_all_valid(m), ("%s: page %p is invalid", __func__, m));