From nobody Mon Nov 29 14:21:02 2021 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 12F1718B01C1; Mon, 29 Nov 2021 14:21:06 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4J2nZl5FDZz3h6H; Mon, 29 Nov 2021 14:21:03 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id C9A6423CD2; Mon, 29 Nov 2021 14:21:02 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 1ATEL2pm055979; Mon, 29 Nov 2021 14:21:02 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 1ATEL2Fe055978; Mon, 29 Nov 2021 14:21:02 GMT (envelope-from git) Date: Mon, 29 Nov 2021 14:21:02 GMT Message-Id: <202111291421.1ATEL2Fe055978@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Mark Johnston Subject: git: cb081566cf86 - stable/13 - vm_page: Consolidate page busy sleep mechanisms List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-branches@freebsd.org X-BeenThere: dev-commits-src-branches@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: markj X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: cb081566cf868dccce2f96bf5276b6dc561c842d Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1638195664; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=7kEuu6Df/fyYZ7axVcIfJ+9c1xjfAMnWAmts5rs7ozs=; b=LPxkO+XHtJZIezBNsXsBLr6gn/0cyM9LfC0NspDXDIWGmmEUxbol4ik8qOtkvtsv+ufkPn fdwoYibCHNBBaA3g87TCx3iRv25//LoQI7jvBx1UJRbgkRmN7cUHvKn5+cQzxIUNlvYZLu FrJbUYqGd/0dIWaJgbhuaDdQBEOudQ81KjEvRHYJMNuOrcTrBkHAApB1kuGDNCfHyk2o4a lGqkXu1crxX0c8r6Pu+hZ0+PNNnNFC008cxsODeCSPDrIAxTJ+1UFiaqW1mB0wpHYObwwB vfHZhc9HeD2n79CWEN5B+no7xRlM/q1R1P5M+AW/MHry6d1fSKMINW4IxQJ6Ug== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1638195664; a=rsa-sha256; cv=none; b=RhLyudshWtvWr7aBh6LO36+1diXgEU8j4VKQ9LItHvGuWIL2+/umewzef7YWv73N7QH2gQ hpI7Gp+16g8AeDjAhCZJCKNgSI+YkJ2rkci5jPQk+Yn515z93ndwQCBj0asrZZDhNbc6pg 9Kgp8aQUCN3tHcKUQN/tR4JENOHotTMJTC7qNBwPguN0ivya4/u3NExSoV/a4aglSA6yAA o6sM4KlX8S7+nkCjO7EPzFGouqV365Rp7A7vhiV3BspDAvV6BknUDmwlkqMKuPa6TMD3dV YYfmqIy02CFi4dzyyJmKeQ1mmY6y7XkkO7ms1SDIjeb5dzalK1t7sLJWjXUTkg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch stable/13 has been updated by markj: URL: https://cgit.FreeBSD.org/src/commit/?id=cb081566cf868dccce2f96bf5276b6dc561c842d commit cb081566cf868dccce2f96bf5276b6dc561c842d Author: Mark Johnston AuthorDate: 2021-11-15 16:35:44 +0000 Commit: Mark Johnston CommitDate: 2021-11-29 14:11:29 +0000 vm_page: Consolidate page busy sleep mechanisms - Modify vm_page_busy_sleep() and vm_page_busy_sleep_unlocked() to take a VM_ALLOC_* flag indicating whether to sleep on shared-busy, and fix up callers. - Modify vm_page_busy_sleep() to return a status indicating whether the object lock was dropped, and fix up callers. - Convert callers of vm_page_sleep_if_busy() to use vm_page_busy_sleep() instead. - Remove vm_page_sleep_if_(x)busy(). No functional change intended. Obtained from: jeff (object_concurrency patches) Reviewed by: kib (cherry picked from commit 87b646630c4892e21446cd096bea6bcaecea33ac) --- ObsoleteFiles.inc | 3 ++ share/man/man9/Makefile | 1 - share/man/man9/vm_page_busy.9 | 40 +++++++++--------- sys/vm/vm_fault.c | 5 +-- sys/vm/vm_object.c | 27 ++++++++---- sys/vm/vm_page.c | 97 ++++++++----------------------------------- sys/vm/vm_page.h | 6 +-- 7 files changed, 62 insertions(+), 117 deletions(-) diff --git a/ObsoleteFiles.inc b/ObsoleteFiles.inc index ac85e8b55e26..1448b9076c95 100644 --- a/ObsoleteFiles.inc +++ b/ObsoleteFiles.inc @@ -36,6 +36,9 @@ # xargs -n1 | sort | uniq -d; # done +# 20211115: vm_page_sleep_if_busy removed +OLD_FILES+=share/man/man9/vm_page_sleep_if_busy.9.gz + # 20210923: rename boot(9) to kern_reboot(9) OLD_FILES+=usr/share/man/man9/boot.9.gz diff --git a/share/man/man9/Makefile b/share/man/man9/Makefile index 500cd252a3df..14cdf7470b41 100644 --- a/share/man/man9/Makefile +++ b/share/man/man9/Makefile @@ -2326,7 +2326,6 @@ MLINKS+=vm_page_busy.9 vm_page_busied.9 \ vm_page_busy.9 vm_page_busy_sleep.9 \ vm_page_busy.9 vm_page_sbusied.9 \ vm_page_busy.9 vm_page_sbusy.9 \ - vm_page_busy.9 vm_page_sleep_if_busy.9 \ vm_page_busy.9 vm_page_sunbusy.9 \ vm_page_busy.9 vm_page_trysbusy.9 \ vm_page_busy.9 vm_page_tryxbusy.9 \ diff --git a/share/man/man9/vm_page_busy.9 b/share/man/man9/vm_page_busy.9 index f4e922a8319f..3f08a467bcb1 100644 --- a/share/man/man9/vm_page_busy.9 +++ b/share/man/man9/vm_page_busy.9 @@ -24,7 +24,7 @@ .\" DAMAGE. .\" .\" $FreeBSD$ -.Dd August 07, 2013 +.Dd November 11, 2021 .Dt VM_PAGE_BUSY 9 .Os .Sh NAME @@ -33,7 +33,6 @@ .Nm vm_page_busy_sleep , .Nm vm_page_sbusied , .Nm vm_page_sbusy , -.Nm vm_page_sleep_if_busy , .Nm vm_page_sunbusy , .Nm vm_page_trysbusy , .Nm vm_page_tryxbusy , @@ -52,14 +51,12 @@ .Fn vm_page_busied "vm_page_t m" .Ft void .Fn vm_page_busy_downgrade "vm_page_t m" -.Ft void -.Fn vm_page_busy_sleep "vm_page_t m" "const char *msg" +.Ft bool +.Fn vm_page_busy_sleep "vm_page_t m" "const char *msg" "int allocflags" .Ft int .Fn vm_page_sbusied "vm_page_t m" .Ft void .Fn vm_page_sbusy "vm_page_t m" -.Ft int -.Fn vm_page_sleep_if_busy "vm_page_t m" "const char *msg" .Ft void .Fn vm_page_sunbusy "vm_page_t m" .Ft int @@ -114,8 +111,23 @@ from an exclusive busy state to a shared busy state. .Pp The .Fn vm_page_busy_sleep -function puts the invoking thread to sleep using the appropriate -waitchannels for the busy mechanism. +checks the busy state of the page +.Fa m +and puts the invoking thread to sleep if the page is busy. +The VM object for the page must be locked. +The +.Fa allocflags +parameter must be either +.Dv 0 , +in which case the function will sleep if the page is busied, +or +.Dv VM_ALLOC_IGN_SBUSY , +in which case the function will sleep only if the page is exclusively +busied. +A return value of true indicates that the invoking thread was put to +sleep and that the object was unlocked. +A return value of false indicates that the invoking thread did not sleep +and the object remains locked. The parameter .Fa msg is a string describing the sleep condition for userland tools. @@ -133,18 +145,6 @@ function shared busies .Fa m . .Pp The -.Fn vm_page_sleep_if_busy -function puts the invoking thread to sleep, using the appropriate -waitchannels for the busy mechanism, if -.Fa m . -is busied in either exclusive or shared mode. -If the invoking thread slept a non-zero value is returned, otherwise -0 is returned. -The parameter -.Fa msg -is a string describing the sleep condition for userland tools. -.Pp -The .Fn vm_page_sunbusy function shared unbusies .Fa m . diff --git a/sys/vm/vm_fault.c b/sys/vm/vm_fault.c index c0ac89e4b70f..646f9ddb9017 100644 --- a/sys/vm/vm_fault.c +++ b/sys/vm/vm_fault.c @@ -1290,9 +1290,8 @@ vm_fault_busy_sleep(struct faultstate *fs) } vm_object_pip_wakeup(fs->object); unlock_map(fs); - if (fs->m == vm_page_lookup(fs->object, fs->pindex)) - vm_page_busy_sleep(fs->m, "vmpfw", false); - else + if (fs->m != vm_page_lookup(fs->object, fs->pindex) || + !vm_page_busy_sleep(fs->m, "vmpfw", 0)) VM_OBJECT_WUNLOCK(fs->object); VM_CNT_INC(v_intrans); vm_object_deallocate(fs->first_object); diff --git a/sys/vm/vm_object.c b/sys/vm/vm_object.c index 47595d38137c..c465a2cf76dd 100644 --- a/sys/vm/vm_object.c +++ b/sys/vm/vm_object.c @@ -1397,7 +1397,8 @@ next_page: */ vm_page_aflag_set(tm, PGA_REFERENCED); } - vm_page_busy_sleep(tm, "madvpo", false); + if (!vm_page_busy_sleep(tm, "madvpo", 0)) + VM_OBJECT_WUNLOCK(tobject); goto relookup; } vm_page_advise(tm, advice); @@ -1581,7 +1582,8 @@ retry: */ if (vm_page_tryxbusy(m) == 0) { VM_OBJECT_WUNLOCK(new_object); - vm_page_sleep_if_busy(m, "spltwt"); + if (vm_page_busy_sleep(m, "spltwt", 0)) + VM_OBJECT_WLOCK(orig_object); VM_OBJECT_WLOCK(new_object); goto retry; } @@ -1667,14 +1669,17 @@ vm_object_collapse_scan_wait(vm_object_t object, vm_page_t p) VM_OBJECT_WUNLOCK(object); VM_OBJECT_WUNLOCK(backing_object); vm_radix_wait(); + VM_OBJECT_WLOCK(object); + } else if (p->object == object) { + VM_OBJECT_WUNLOCK(backing_object); + if (vm_page_busy_sleep(p, "vmocol", 0)) + VM_OBJECT_WLOCK(object); } else { - if (p->object == object) + VM_OBJECT_WUNLOCK(object); + if (!vm_page_busy_sleep(p, "vmocol", 0)) VM_OBJECT_WUNLOCK(backing_object); - else - VM_OBJECT_WUNLOCK(object); - vm_page_busy_sleep(p, "vmocol", false); + VM_OBJECT_WLOCK(object); } - VM_OBJECT_WLOCK(object); VM_OBJECT_WLOCK(backing_object); return (TAILQ_FIRST(&backing_object->memq)); } @@ -2118,7 +2123,8 @@ again: * not specified. */ if (vm_page_tryxbusy(p) == 0) { - vm_page_sleep_if_busy(p, "vmopar"); + if (vm_page_busy_sleep(p, "vmopar", 0)) + VM_OBJECT_WLOCK(object); goto again; } if ((options & OBJPR_VALIDONLY) != 0 && vm_page_none_valid(p)) { @@ -2426,7 +2432,10 @@ again: VM_OBJECT_RUNLOCK(tobject); tobject = t1object; } - vm_page_busy_sleep(tm, "unwbo", true); + tobject = tm->object; + if (!vm_page_busy_sleep(tm, "unwbo", + VM_ALLOC_IGN_SBUSY)) + VM_OBJECT_RUNLOCK(tobject); goto again; } vm_page_unwire(tm, queue); diff --git a/sys/vm/vm_page.c b/sys/vm/vm_page.c index 6be310531924..74c03ed3a21b 100644 --- a/sys/vm/vm_page.c +++ b/sys/vm/vm_page.c @@ -1000,24 +1000,26 @@ vm_page_sunbusy(vm_page_t m) * vm_page_busy_sleep: * * Sleep if the page is busy, using the page pointer as wchan. - * This is used to implement the hard-path of busying mechanism. + * This is used to implement the hard-path of the busying mechanism. + * + * If VM_ALLOC_IGN_SBUSY is specified in allocflags, the function + * will not sleep if the page is shared-busy. * - * If nonshared is true, sleep only if the page is xbusy. + * The object lock must be held on entry. * - * The object lock must be held on entry and will be released on exit. + * Returns true if it slept and dropped the object lock, or false + * if there was no sleep and the lock is still held. */ -void -vm_page_busy_sleep(vm_page_t m, const char *wmesg, bool nonshared) +bool +vm_page_busy_sleep(vm_page_t m, const char *wmesg, int allocflags) { vm_object_t obj; obj = m->object; VM_OBJECT_ASSERT_LOCKED(obj); - vm_page_lock_assert(m, MA_NOTOWNED); - if (!_vm_page_busy_sleep(obj, m, m->pindex, wmesg, - nonshared ? VM_ALLOC_SBUSY : 0 , true)) - VM_OBJECT_DROP(obj); + return (_vm_page_busy_sleep(obj, m, m->pindex, wmesg, allocflags, + true)); } /* @@ -1026,21 +1028,19 @@ vm_page_busy_sleep(vm_page_t m, const char *wmesg, bool nonshared) * Sleep if the page is busy, using the page pointer as wchan. * This is used to implement the hard-path of busying mechanism. * - * If nonshared is true, sleep only if the page is xbusy. + * If VM_ALLOC_IGN_SBUSY is specified in allocflags, the function + * will not sleep if the page is shared-busy. * * The object lock must not be held on entry. The operation will * return if the page changes identity. */ void vm_page_busy_sleep_unlocked(vm_object_t obj, vm_page_t m, vm_pindex_t pindex, - const char *wmesg, bool nonshared) + const char *wmesg, int allocflags) { - VM_OBJECT_ASSERT_UNLOCKED(obj); - vm_page_lock_assert(m, MA_NOTOWNED); - _vm_page_busy_sleep(obj, m, pindex, wmesg, - nonshared ? VM_ALLOC_SBUSY : 0, false); + (void)_vm_page_busy_sleep(obj, m, pindex, wmesg, allocflags, false); } /* @@ -1050,6 +1050,8 @@ vm_page_busy_sleep_unlocked(vm_object_t obj, vm_page_t m, vm_pindex_t pindex, * lockstate against parameters. Returns true if it sleeps and * false otherwise. * + * allocflags uses VM_ALLOC_* flags to specify the lock required. + * * If locked is true the lock will be dropped for any true returns * and held for any false returns. */ @@ -1395,71 +1397,6 @@ vm_page_free_invalid(vm_page_t m) vm_page_free(m); } -/* - * vm_page_sleep_if_busy: - * - * Sleep and release the object lock if the page is busied. - * Returns TRUE if the thread slept. - * - * The given page must be unlocked and object containing it must - * be locked. - */ -int -vm_page_sleep_if_busy(vm_page_t m, const char *wmesg) -{ - vm_object_t obj; - - vm_page_lock_assert(m, MA_NOTOWNED); - VM_OBJECT_ASSERT_WLOCKED(m->object); - - /* - * The page-specific object must be cached because page - * identity can change during the sleep, causing the - * re-lock of a different object. - * It is assumed that a reference to the object is already - * held by the callers. - */ - obj = m->object; - if (_vm_page_busy_sleep(obj, m, m->pindex, wmesg, 0, true)) { - VM_OBJECT_WLOCK(obj); - return (TRUE); - } - return (FALSE); -} - -/* - * vm_page_sleep_if_xbusy: - * - * Sleep and release the object lock if the page is xbusied. - * Returns TRUE if the thread slept. - * - * The given page must be unlocked and object containing it must - * be locked. - */ -int -vm_page_sleep_if_xbusy(vm_page_t m, const char *wmesg) -{ - vm_object_t obj; - - vm_page_lock_assert(m, MA_NOTOWNED); - VM_OBJECT_ASSERT_WLOCKED(m->object); - - /* - * The page-specific object must be cached because page - * identity can change during the sleep, causing the - * re-lock of a different object. - * It is assumed that a reference to the object is already - * held by the callers. - */ - obj = m->object; - if (_vm_page_busy_sleep(obj, m, m->pindex, wmesg, VM_ALLOC_SBUSY, - true)) { - VM_OBJECT_WLOCK(obj); - return (TRUE); - } - return (FALSE); -} - /* * vm_page_dirty_KBI: [ internal use only ] * diff --git a/sys/vm/vm_page.h b/sys/vm/vm_page.h index b31ea0241cc7..4b43b1d71f54 100644 --- a/sys/vm/vm_page.h +++ b/sys/vm/vm_page.h @@ -592,9 +592,9 @@ malloc2vm_flags(int malloc_flags) bool vm_page_busy_acquire(vm_page_t m, int allocflags); void vm_page_busy_downgrade(vm_page_t m); int vm_page_busy_tryupgrade(vm_page_t m); -void vm_page_busy_sleep(vm_page_t m, const char *msg, bool nonshared); +bool vm_page_busy_sleep(vm_page_t m, const char *msg, int allocflags); void vm_page_busy_sleep_unlocked(vm_object_t obj, vm_page_t m, - vm_pindex_t pindex, const char *wmesg, bool nonshared); + vm_pindex_t pindex, const char *wmesg, int allocflags); void vm_page_free(vm_page_t m); void vm_page_free_zero(vm_page_t m); @@ -678,8 +678,6 @@ vm_page_t vm_page_scan_contig(u_long npages, vm_page_t m_start, vm_page_t m_end, u_long alignment, vm_paddr_t boundary, int options); vm_page_bits_t vm_page_set_dirty(vm_page_t m); void vm_page_set_valid_range(vm_page_t m, int base, int size); -int vm_page_sleep_if_busy(vm_page_t m, const char *msg); -int vm_page_sleep_if_xbusy(vm_page_t m, const char *msg); vm_offset_t vm_page_startup(vm_offset_t vaddr); void vm_page_sunbusy(vm_page_t m); bool vm_page_try_remove_all(vm_page_t m);