From nobody Sat Mar 25 22:16:36 2023 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PkYMS4FgCz41nXs; Sat, 25 Mar 2023 22:16:36 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PkYMS3jQxz45fQ; Sat, 25 Mar 2023 22:16:36 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679782596; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=ZUA4MSoYUGBC1Wz2yoyAPJrWjfMRR9L0UWiyve0l6/I=; b=rfZP80BaqvzwHLpyFOw4QPW1zrl4SHhHi0ouDMjqafu4DzDBV0EuHvTJFzxCXoLH2zaR3W Uq9YdBuncDXCIq7EGDKXRqlRaxiUu7yRgm+yzvf6ifjmH4A+Ka39Mzi9tmTHo+Cd7Ba702 LtAKVdeXGiMeopQecjplcf4IWJm4PUJr+taNgJ8NtEXptYVoPZEsUfyozXX5/JlSmcVuys 00+w1m3ZcCcgGqVdZydko7wrBeTc0Q6RKRojLw7+x4kVkFMuOGlQ/YIbe8Rc91/InFK/fa fyMk4DdvZEVt9Phe3tndjG16v1Sxv5AsvHbxFwopHlWrciUFHsAt8mx5IAbbog== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679782596; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=ZUA4MSoYUGBC1Wz2yoyAPJrWjfMRR9L0UWiyve0l6/I=; b=KQGbsvZ6F0ct8aQMcHctyf307sozxGgcDFZIRNCghyLKoCQH4SiOZnQguE7O2WdZoOVTpN S2adsle4wNgszOo2E4QSr6oMAnW0cz1aoyiYJX/+bJyv6ofX++hrQVoonPTw0Itsp9eZpJ b1O2yR3Z3bdMw7efBiDfJeH/W7WKx5Tltb0neJsb05TxJb9kzXWX5A4gdRcS+P7PBv3pYI io+IxUJd7DQ0G4hVVo2bO9fcwqCkk3H4vQKPnjtRjVHNCWQfheDOkEw6xfr0v2CU9q8rFT vybVpfqojHObBlEekrZJ+sS5StC+kkRq2E5If5Vvjh2ke74GG+EMqznWjr3tCA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1679782596; a=rsa-sha256; cv=none; b=xpBRtC2f1sDIxnLqQvxy5m1HEfYht4uE/7QGyNOm3R3O3SF/0ltUy7zMkAfPrMTsdUrhuN W42ODaDcbhRhmPtgRtYgKHjHM8lF3ALTO3nqM4iqUppA1JnJ6KYTFsKVHobmx2tEbZ5Uxj 2yBsNQWEv7lvTrvZkbC+HDCF7gJhGAc5cSbPzpDXwncwMmAhdKfdxajHrhA+NjZAZLe2OG JfmMVSUkkZDLJmrNTlIctX75Q3vlwCXV5E6W2Eqjixav/sbJqwtCp7PWE7BBWuMvoIEhsp +UqsDQC7ynVWcAE4B3v57CrprwS7TPZsFNv/+EDAhVyfqdVuq82Wvcds3MF7Eg== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4PkYMS2bjvzGjV; Sat, 25 Mar 2023 22:16:36 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 32PMGaN9089134; Sat, 25 Mar 2023 22:16:36 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 32PMGavs089133; Sat, 25 Mar 2023 22:16:36 GMT (envelope-from git) Date: Sat, 25 Mar 2023 22:16:36 GMT Message-Id: <202303252216.32PMGavs089133@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Mateusz Guzik Subject: git: 0e71f4f77c01 - main - vm: add unlocked page lookup before trying vm_fault_soft_fast List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: mjg X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 0e71f4f77c016b4087106e7c58b958667df8e1b2 Auto-Submitted: auto-generated X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by mjg: URL: https://cgit.FreeBSD.org/src/commit/?id=0e71f4f77c016b4087106e7c58b958667df8e1b2 commit 0e71f4f77c016b4087106e7c58b958667df8e1b2 Author: Mateusz Guzik AuthorDate: 2023-03-25 14:40:55 +0000 Commit: Mateusz Guzik CommitDate: 2023-03-25 22:14:59 +0000 vm: add unlocked page lookup before trying vm_fault_soft_fast Shaves a read lock + tryupgrade trip most of the time. Stats from doing a kernel build (counters not present in the tree): vm.fault_soft_fast_ok: 262653 vm.fault_soft_fast_failed_other: 41 vm.fault_soft_fast_failed_no_page: 39595772 vm.fault_soft_fast_failed_page_busy: 1929 vm.fault_soft_fast_failed_page_invalid: 22183 Reviewed by: kib Differential Revision: https://reviews.freebsd.org/D39268 --- sys/vm/vm_fault.c | 52 ++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 40 insertions(+), 12 deletions(-) diff --git a/sys/vm/vm_fault.c b/sys/vm/vm_fault.c index 056e6e12e8c8..1f9063a5988a 100644 --- a/sys/vm/vm_fault.c +++ b/sys/vm/vm_fault.c @@ -348,13 +348,38 @@ vm_fault_soft_fast(struct faultstate *fs) MPASS(fs->vp == NULL); + /* + * If we fail, vast majority of the time it is because the page is not + * there to begin with. Opportunistically perform the lookup and + * subsequent checks without the object lock, revalidate later. + * + * Note: a busy page can be mapped for read|execute access. + */ + m = vm_page_lookup_unlocked(fs->first_object, fs->first_pindex); + if (m == NULL || !vm_page_all_valid(m) || + ((fs->prot & VM_PROT_WRITE) != 0 && vm_page_busied(m))) { + VM_OBJECT_WLOCK(fs->first_object); + return (FAULT_FAILURE); + } + vaddr = fs->vaddr; - vm_object_busy(fs->first_object); - m = vm_page_lookup(fs->first_object, fs->first_pindex); - /* A busy page can be mapped for read|execute access. */ - if (m == NULL || ((fs->prot & VM_PROT_WRITE) != 0 && - vm_page_busied(m)) || !vm_page_all_valid(m)) + + VM_OBJECT_RLOCK(fs->first_object); + + /* + * Now that we stabilized the state, revalidate the page is in the shape + * we encountered above. + */ + + if (m->object != fs->first_object || m->pindex != fs->first_pindex) goto fail; + + vm_object_busy(fs->first_object); + + if (!vm_page_all_valid(m) || + ((fs->prot & VM_PROT_WRITE) != 0 && vm_page_busied(m))) + goto fail_busy; + m_map = m; psind = 0; #if VM_NRESERVLEVEL > 0 @@ -390,7 +415,7 @@ vm_fault_soft_fast(struct faultstate *fs) if (pmap_enter(fs->map->pmap, vaddr, m_map, fs->prot, fs->fault_type | PMAP_ENTER_NOSLEEP | (fs->wired ? PMAP_ENTER_WIRED : 0), psind) != KERN_SUCCESS) - goto fail; + goto fail_busy; if (fs->m_hold != NULL) { (*fs->m_hold) = m; vm_page_wire(m); @@ -403,8 +428,13 @@ vm_fault_soft_fast(struct faultstate *fs) vm_map_lookup_done(fs->map, fs->entry); curthread->td_ru.ru_minflt++; return (FAULT_SUCCESS); -fail: +fail_busy: vm_object_unbusy(fs->first_object); +fail: + if (!VM_OBJECT_TRYUPGRADE(fs->first_object)) { + VM_OBJECT_RUNLOCK(fs->first_object); + VM_OBJECT_WLOCK(fs->first_object); + } return (FAULT_FAILURE); } @@ -1556,14 +1586,12 @@ RetryFault: if (fs.vp == NULL /* avoid locked vnode leak */ && (fs.entry->eflags & MAP_ENTRY_SPLIT_BOUNDARY_MASK) == 0 && (fs.fault_flags & (VM_FAULT_WIRE | VM_FAULT_DIRTY)) == 0) { - VM_OBJECT_RLOCK(fs.first_object); res = vm_fault_soft_fast(&fs); - if (res == FAULT_SUCCESS) + if (res == FAULT_SUCCESS) { + VM_OBJECT_ASSERT_UNLOCKED(fs.first_object); return (KERN_SUCCESS); - if (!VM_OBJECT_TRYUPGRADE(fs.first_object)) { - VM_OBJECT_RUNLOCK(fs.first_object); - VM_OBJECT_WLOCK(fs.first_object); } + VM_OBJECT_ASSERT_WLOCKED(fs.first_object); } else { VM_OBJECT_WLOCK(fs.first_object); }