From nobody Thu Mar 31 06:51:54 2022 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 195F11A44AE8; Thu, 31 Mar 2022 06:51:55 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KTYrC07YBz4S07; Thu, 31 Mar 2022 06:51:55 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1648709515; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=mfJ4hDSoHusIq7unMwAFbBX31i++Ni5zxByP2zDZlwI=; b=fjtpY0vrWJGeb/a1Tgw2OMduesA7W6l7dHeAB1epOhdREdd5lsylg1N+dWQeWCSxfDktoN bcO5dO6OICFoy0DSCk63Jsw7KA9lRRuad5Zyhmccv4vt+sxTxkBmcKjCCiJE2bRJdgy6hD E/0h/SR3k5h+USUMrgO8SqhNIfvQ/Su/pfIfCEC8loO5WdAwwqRsowIlPbBcDBRM+wXrbr yoHfFP9ZErhZwyKBK8KgIngx91LCLqpLSx5uS7+nrm7YbTFDL3V3T74oToH0C7FqYo9NAY Y5Yj2LejTk83JKp4ZhF1SwLiCCtKXIUckGmgZIYFG9X7j/AKL2kBdxa5zOSF7Q== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id DA002227D7; Thu, 31 Mar 2022 06:51:54 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 22V6psok049770; Thu, 31 Mar 2022 06:51:54 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 22V6psTx049769; Thu, 31 Mar 2022 06:51:54 GMT (envelope-from git) Date: Thu, 31 Mar 2022 06:51:54 GMT Message-Id: <202203310651.22V6psTx049769@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Doug Moore Subject: git: 342056fa1c7a - main - vm_phys: alloc pages without duplicating searches. List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: dougm X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 342056fa1c7a36d90feafd593fb980f98563f32c Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1648709515; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=mfJ4hDSoHusIq7unMwAFbBX31i++Ni5zxByP2zDZlwI=; b=Kkdc3F1ETd3ukLxmCMDJ/VOaX8Q7YpmPB3jPlvt6mFZ1P4t3MNEteGP3zkYHWPF2PTX3ga A+PlT0UF8Sy/Or2WIf4IOUYT8bz7xG51euGUKyM63xTzdRj5uPXTTBeuO1ShLmnCFmzCBG qEZo5mILMkItZvMQgzV0a8TbX9J/ca5tBHmnfAefJie+uk419HPiQzD50BTZMb0LY+MqQg Fv8zk95FJ5Y2AEgTdp+sdbkDSSJpmT8ldEDM7iTK31bTO0f5IbamC4uZKZSma7H+d0ly6U fnyL+1EVAfZf4T+TY3kdyX5+PGBEYZ0vSlMrkrLH5Pu+Gun2riW07mDbjhujwQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1648709515; a=rsa-sha256; cv=none; b=LvSe33oXQo7L9cDu5ORoHbYzn3BTcps4VxWSLQaFSezdu4Q8/oyWRPYDwB4l8IN+iY5L9A ylFQ8JYvhpZ/QP2K36WQChICcqjpNdsrWCVhdmE3cyeoOkGxcXhxdTN12u3wAkmlNt6Tz9 cB7tgbnixWrhjK1Nx8frMWfWYreAm0xFEMl5DNpxUJVBD3bgbGTyBS318JamnH4IDfUuU7 HlZOVuzzPpMN7e5G7Hggq5xMb+DNRd6L6F7dpI/OGANHnHovS3eYs0FhnyuLEmJCWZrmHq I56hmdyPcOr0lkyGxmRjXizZ49lKn8qSGEgFxCtEewl3d0FDpS5z8YLscEigRA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by dougm: URL: https://cgit.FreeBSD.org/src/commit/?id=342056fa1c7a36d90feafd593fb980f98563f32c commit 342056fa1c7a36d90feafd593fb980f98563f32c Author: Doug Moore AuthorDate: 2022-03-31 06:40:46 +0000 Commit: Doug Moore CommitDate: 2022-03-31 06:40:46 +0000 vm_phys: alloc pages without duplicating searches. In the search for contiguous pages, as each page segment is examined, check to see if the free list set for the next page segment differs from the set for the current segment, and avoid a pointless search if they do not differ. Discussed with: alc Reviewed by: markj Tested by: pho Differential Revision: https://reviews.freebsd.org/D33947 --- sys/vm/vm_phys.c | 134 ++++++++++++++++++++++++++++++------------------------- 1 file changed, 72 insertions(+), 62 deletions(-) diff --git a/sys/vm/vm_phys.c b/sys/vm/vm_phys.c index 7ca3f5fbc0c5..1f4c228b1a84 100644 --- a/sys/vm/vm_phys.c +++ b/sys/vm/vm_phys.c @@ -173,9 +173,6 @@ SYSCTL_OID(_vm, OID_AUTO, phys_locality, SYSCTL_INT(_vm, OID_AUTO, ndomains, CTLFLAG_RD, &vm_ndomains, 0, "Number of physical memory domains available."); -static vm_page_t vm_phys_alloc_seg_contig(struct vm_phys_seg *seg, - u_long npages, vm_paddr_t low, vm_paddr_t high, u_long alignment, - vm_paddr_t boundary); static void _vm_phys_create_seg(vm_paddr_t start, vm_paddr_t end, int domain); static void vm_phys_create_seg(vm_paddr_t start, vm_paddr_t end); static void vm_phys_split_pages(vm_page_t m, int oind, struct vm_freelist *fl, @@ -1351,63 +1348,16 @@ vm_phys_unfree_page(vm_page_t m) } /* - * Allocate a contiguous set of physical pages of the given size - * "npages" from the free lists. All of the physical pages must be at - * or above the given physical address "low" and below the given - * physical address "high". The given value "alignment" determines the - * alignment of the first physical page in the set. If the given value - * "boundary" is non-zero, then the set of physical pages cannot cross - * any physical address boundary that is a multiple of that value. Both - * "alignment" and "boundary" must be a power of two. + * Allocate a run of contiguous physical pages from the specified free list + * table. */ -vm_page_t -vm_phys_alloc_contig(int domain, u_long npages, vm_paddr_t low, vm_paddr_t high, +static vm_page_t +vm_phys_alloc_queues_contig( + struct vm_freelist (*queues)[VM_NFREEPOOL][VM_NFREEORDER_MAX], + u_long npages, vm_paddr_t low, vm_paddr_t high, u_long alignment, vm_paddr_t boundary) { - vm_paddr_t pa_end, pa_start; - vm_page_t m_run; struct vm_phys_seg *seg; - int segind; - - KASSERT(npages > 0, ("npages is 0")); - KASSERT(powerof2(alignment), ("alignment is not a power of 2")); - KASSERT(powerof2(boundary), ("boundary is not a power of 2")); - vm_domain_free_assert_locked(VM_DOMAIN(domain)); - if (low >= high) - return (NULL); - m_run = NULL; - for (segind = vm_phys_nsegs - 1; segind >= 0; segind--) { - seg = &vm_phys_segs[segind]; - if (seg->start >= high || seg->domain != domain) - continue; - if (low >= seg->end) - break; - if (low <= seg->start) - pa_start = seg->start; - else - pa_start = low; - if (high < seg->end) - pa_end = high; - else - pa_end = seg->end; - if (pa_end - pa_start < ptoa(npages)) - continue; - m_run = vm_phys_alloc_seg_contig(seg, npages, low, high, - alignment, boundary); - if (m_run != NULL) - break; - } - return (m_run); -} - -/* - * Allocate a run of contiguous physical pages from the free list for the - * specified segment. - */ -static vm_page_t -vm_phys_alloc_seg_contig(struct vm_phys_seg *seg, u_long npages, - vm_paddr_t low, vm_paddr_t high, u_long alignment, vm_paddr_t boundary) -{ struct vm_freelist *fl; vm_paddr_t pa, pa_end, size; vm_page_t m, m_ret; @@ -1417,7 +1367,6 @@ vm_phys_alloc_seg_contig(struct vm_phys_seg *seg, u_long npages, KASSERT(npages > 0, ("npages is 0")); KASSERT(powerof2(alignment), ("alignment is not a power of 2")); KASSERT(powerof2(boundary), ("boundary is not a power of 2")); - vm_domain_free_assert_locked(VM_DOMAIN(seg->domain)); /* Compute the queue that is the best fit for npages. */ order = flsl(npages - 1); /* Search for a run satisfying the specified conditions. */ @@ -1425,7 +1374,7 @@ vm_phys_alloc_seg_contig(struct vm_phys_seg *seg, u_long npages, for (oind = min(order, VM_NFREEORDER - 1); oind < VM_NFREEORDER; oind++) { for (pind = 0; pind < VM_NFREEPOOL; pind++) { - fl = (*seg->free_queues)[pind]; + fl = (*queues)[pind]; TAILQ_FOREACH(m_ret, &fl[oind].pl, listq) { /* * Determine if the address range starting at pa @@ -1451,8 +1400,8 @@ vm_phys_alloc_seg_contig(struct vm_phys_seg *seg, u_long npages, * (without overflow in pa_end calculation) * and fits within the segment. */ - if (pa_end < pa || - pa < seg->start || seg->end < pa_end) + seg = &vm_phys_segs[m_ret->segind]; + if (pa_end < pa || seg->end < pa_end) continue; /* @@ -1473,7 +1422,7 @@ vm_phys_alloc_seg_contig(struct vm_phys_seg *seg, u_long npages, return (NULL); done: for (m = m_ret; m < &m_ret[npages]; m = &m[1 << oind]) { - fl = (*seg->free_queues)[m->pool]; + fl = (*queues)[m->pool]; vm_freelist_rem(fl, m, oind); if (m->pool != VM_FREEPOOL_DEFAULT) vm_phys_set_pool(VM_FREEPOOL_DEFAULT, m, oind); @@ -1481,12 +1430,73 @@ done: /* Return excess pages to the free lists. */ npages_end = roundup2(npages, 1 << oind); if (npages < npages_end) { - fl = (*seg->free_queues)[VM_FREEPOOL_DEFAULT]; + fl = (*queues)[VM_FREEPOOL_DEFAULT]; vm_phys_enq_range(&m_ret[npages], npages_end - npages, fl, 0); } return (m_ret); } +/* + * Allocate a contiguous set of physical pages of the given size + * "npages" from the free lists. All of the physical pages must be at + * or above the given physical address "low" and below the given + * physical address "high". The given value "alignment" determines the + * alignment of the first physical page in the set. If the given value + * "boundary" is non-zero, then the set of physical pages cannot cross + * any physical address boundary that is a multiple of that value. Both + * "alignment" and "boundary" must be a power of two. + */ +vm_page_t +vm_phys_alloc_contig(int domain, u_long npages, vm_paddr_t low, vm_paddr_t high, + u_long alignment, vm_paddr_t boundary) +{ + vm_paddr_t pa_end, pa_start; + vm_page_t m_run; + struct vm_phys_seg *seg; + struct vm_freelist (*queues)[VM_NFREEPOOL][VM_NFREEORDER_MAX]; + int segind; + + KASSERT(npages > 0, ("npages is 0")); + KASSERT(powerof2(alignment), ("alignment is not a power of 2")); + KASSERT(powerof2(boundary), ("boundary is not a power of 2")); + vm_domain_free_assert_locked(VM_DOMAIN(domain)); + if (low >= high) + return (NULL); + queues = NULL; + m_run = NULL; + for (segind = vm_phys_nsegs - 1; segind >= 0; segind--) { + seg = &vm_phys_segs[segind]; + if (seg->start >= high || seg->domain != domain) + continue; + if (low >= seg->end) + break; + if (low <= seg->start) + pa_start = seg->start; + else + pa_start = low; + if (high < seg->end) + pa_end = high; + else + pa_end = seg->end; + if (pa_end - pa_start < ptoa(npages)) + continue; + /* + * If a previous segment led to a search using + * the same free lists as would this segment, then + * we've actually already searched within this + * too. So skip it. + */ + if (seg->free_queues == queues) + continue; + queues = seg->free_queues; + m_run = vm_phys_alloc_queues_contig(queues, npages, + low, high, alignment, boundary); + if (m_run != NULL) + break; + } + return (m_run); +} + /* * Return the index of the first unused slot which may be the terminating * entry.