From nobody Thu Apr 10 07:20:23 2025 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZYB6c2H3fz5sZkN; Thu, 10 Apr 2025 07:20:24 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZYB6c106Zz3K7V; Thu, 10 Apr 2025 07:20:24 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1744269624; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=hkBvHEjuzmLPyAi+qBzgcC8bL6YiqGQrKTBevQEoMUE=; b=teUgXYmthm469L46+A/8gf3wIc7qw7SqBxFGIPWzwzVjs7udsR2kpMW5KpslSTK6JKaphD hrgc9A2FnOiKm5bsqL/CJWFJg7Na7kZO7nppOfye4NBk6uIqmtamJIzAYsuDmHLWDyF7YG I544iqiqGu41CK43m4qKetQ000weCUNh1nO/Zx/XgH5h5d0Z10h8tdUUgPI3WN2ENsF26J Q25Lht41CxNnQVUEYBeY2qHxujFwnbNZABvI6wb1kmJDVfTI3Pi8K8wQFXcuCKHN61zqMh GnMWNdUVS+1sfEIBG9xkESPk1GAsN3py2jNle7Bl/4Kxtt8QAVKWhQct8e81Zg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1744269624; a=rsa-sha256; cv=none; b=UH+aODmp2wkYlKLrQeMJEvmFc39DBmbDcZXGTJJJglqaVsy4nn94vaMmnnCgfmNL55rKcf UKkZsN4dUnYiiNDgDjeIPIUMOsCeppvTyAqhUeLDTqVHpAmiO0wCESoXnVtFOikVyxp4KR uGYjgDLxiOPDZfy0U75dPCKfmStgXg1HRHwJ155IIv5QyMDO1rOGO2IwV0RkMz//rdmqUO B44BNt+PFMPLxYp7Laz1/bkHewDoRr7WD9WNClsSYyFeM0l36U9ftrn0oABvgYcEjcJbAi pEz++3ANmZg7Jeh1BOcuC7GRdzSVvzUJfF1sCAr/Q6gPQc1e6fCyP79GseRyiw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1744269624; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=hkBvHEjuzmLPyAi+qBzgcC8bL6YiqGQrKTBevQEoMUE=; b=JnwPqvoidn871I8t5xrA9c1C/SlpTAV0Rnvmcc7Y6Z82gfhJUP3a9tGnwVET7iO3qvyhjh /soDoIp+ZV+sd6eRBOhx+WkY3iRWsiaoDrRaxKdE0AT7A/7+9SjBRPuh360BsihdXThZTP bGEGSsu33IpPcPjnDzhube8C6w6MdBfABoYKgziSIfFeS4bO1zj8Vf7Q0zK30HxIjPiFc3 tQY7/6q27wVvoLbLv+TgJmpf6EgD8kkOX/CCfRNTekWDxZYywt/HzFEVFSzkZmXAPq3h5x kDfq4kN/KTQcYizmApeKABXxmd3L3EMQAafikDYsOdg4xnTN23oGXQr8tSuy8g== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZYB6b6rczzsvg; Thu, 10 Apr 2025 07:20:23 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 53A7KNmF009280; Thu, 10 Apr 2025 07:20:23 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 53A7KNh2009277; Thu, 10 Apr 2025 07:20:23 GMT (envelope-from git) Date: Thu, 10 Apr 2025 07:20:23 GMT Message-Id: <202504100720.53A7KNh2009277@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Doug Moore Subject: git: f06cbfc5592e - main - vm_fault: use iterator for allocation List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: dougm X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: f06cbfc5592e58104fe01f3f458928ee09be24fc Auto-Submitted: auto-generated The branch main has been updated by dougm: URL: https://cgit.FreeBSD.org/src/commit/?id=f06cbfc5592e58104fe01f3f458928ee09be24fc commit f06cbfc5592e58104fe01f3f458928ee09be24fc Author: Doug Moore AuthorDate: 2025-04-10 07:19:06 +0000 Commit: Doug Moore CommitDate: 2025-04-10 07:19:06 +0000 vm_fault: use iterator for allocation Pass an iterator to vm_fault_allocate, so that it can invoke vm_page_alloc_after. In vm_fault_object, use an iterator for a page lookup so that the search can benefit the allocation in vm_fault_allocate. In vm_fault, define an iterator, initialize it only when it might be used, and pass it on to vm_fault_allocate. Reviewed by: kib Differential Revision: https://reviews.freebsd.org/D49741 --- sys/vm/vm_fault.c | 18 ++++++++++++------ sys/vm/vm_radix.h | 13 +++++++++++++ 2 files changed, 25 insertions(+), 6 deletions(-) diff --git a/sys/vm/vm_fault.c b/sys/vm/vm_fault.c index 48a7a47e4c59..81631b672040 100644 --- a/sys/vm/vm_fault.c +++ b/sys/vm/vm_fault.c @@ -105,6 +105,7 @@ #include #include #include +#include #include #include @@ -1230,7 +1231,7 @@ vm_fault_allocate_oom(struct faultstate *fs) * Allocate a page directly or via the object populate method. */ static enum fault_status -vm_fault_allocate(struct faultstate *fs) +vm_fault_allocate(struct faultstate *fs, struct pctrie_iter *pages) { struct domainset *dset; enum fault_status res; @@ -1291,8 +1292,9 @@ vm_fault_allocate(struct faultstate *fs) vm_fault_unlock_and_deallocate(fs); return (FAULT_FAILURE); } - fs->m = vm_page_alloc(fs->object, fs->pindex, - P_KILLED(curproc) ? VM_ALLOC_SYSTEM : 0); + fs->m = vm_page_alloc_after(fs->object, fs->pindex, + P_KILLED(curproc) ? VM_ALLOC_SYSTEM : 0, + vm_radix_iter_lookup_lt(pages, fs->pindex)); } if (fs->m == NULL) { if (vm_fault_allocate_oom(fs)) @@ -1459,6 +1461,7 @@ vm_fault_busy_sleep(struct faultstate *fs) static enum fault_status vm_fault_object(struct faultstate *fs, int *behindp, int *aheadp) { + struct pctrie_iter pages; enum fault_status res; bool dead; @@ -1484,7 +1487,8 @@ vm_fault_object(struct faultstate *fs, int *behindp, int *aheadp) /* * See if the page is resident. */ - fs->m = vm_page_lookup(fs->object, fs->pindex); + vm_page_iter_init(&pages, fs->object); + fs->m = vm_radix_iter_lookup(&pages, fs->pindex); if (fs->m != NULL) { if (!vm_page_tryxbusy(fs->m)) { vm_fault_busy_sleep(fs); @@ -1514,7 +1518,7 @@ vm_fault_object(struct faultstate *fs, int *behindp, int *aheadp) vm_fault_unlock_and_deallocate(fs); return (FAULT_RESTART); } - res = vm_fault_allocate(fs); + res = vm_fault_allocate(fs, &pages); if (res != FAULT_CONTINUE) return (res); } @@ -1549,6 +1553,7 @@ int vm_fault(vm_map_t map, vm_offset_t vaddr, vm_prot_t fault_type, int fault_flags, vm_page_t *m_hold) { + struct pctrie_iter pages; struct faultstate fs; int ahead, behind, faultcount, rv; enum fault_status res; @@ -1603,6 +1608,7 @@ RetryFault: } VM_OBJECT_ASSERT_WLOCKED(fs.first_object); } else { + vm_page_iter_init(&pages, fs.first_object); VM_OBJECT_WLOCK(fs.first_object); } @@ -1627,7 +1633,7 @@ RetryFault: fs.pindex = fs.first_pindex; if ((fs.entry->eflags & MAP_ENTRY_SPLIT_BOUNDARY_MASK) != 0) { - res = vm_fault_allocate(&fs); + res = vm_fault_allocate(&fs, &pages); switch (res) { case FAULT_RESTART: goto RetryFault; diff --git a/sys/vm/vm_radix.h b/sys/vm/vm_radix.h index 561e36f2e164..e9ee3e7f3911 100644 --- a/sys/vm/vm_radix.h +++ b/sys/vm/vm_radix.h @@ -270,6 +270,19 @@ vm_radix_iter_lookup_le(struct pctrie_iter *pages, vm_pindex_t index) return (VM_RADIX_PCTRIE_ITER_LOOKUP_LE(pages, index)); } +/* + * Initialize an iterator pointing to the page with the greatest pindex that is + * less than to the specified pindex, or NULL if there are no such + * pages. Return the page. + * + * Requires that access be externally synchronized by a lock. + */ +static __inline vm_page_t +vm_radix_iter_lookup_lt(struct pctrie_iter *pages, vm_pindex_t index) +{ + return (index == 0 ? NULL : vm_radix_iter_lookup_le(pages, index - 1)); +} + /* * Update the iterator to point to the page with the pindex that is one greater * than the current pindex, or NULL if there is no such page. Return the page.