From nobody Wed Feb 09 00:43:34 2022 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 1E90419B157A; Wed, 9 Feb 2022 00:43:35 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Jth2G5BX6z4pmT; Wed, 9 Feb 2022 00:43:34 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1644367415; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=FrkJnph1Yqo5f4Os5hmUchvhTPoP6oF5+YkyGjx+304=; b=EuvGC/rHrxtKJFcB4PN2Xuorq/tT5kVr7J8DbF+5ZBPOtN3pwj0j3d7akCo6Za547cQIOU ZQWvCFW2Jr5iMjvW5waXXQTJ6xX0KQZfC4MQKV2cuFqvVwxt+VE+zYEyv3B2ILEcinBfwT qaU5YmKLokokxvC8gmCRKFSgoHqAElBeOnnxS+ss4/q0hC4y3eZzdWNVYWew4EzStqcjM2 2xPy/wLr3jxbdrRmFMIp0ZwrXCBo3A/0Nyk7QccbA2Pn0m2Nwn3FXCHpD6f2sWtYAqiDHh wFYCnySxvZTMbLkLgzIbLPMWGvgjGFezRhK1opL+mZ0dJiS6fjRNM09IhQOFIw== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 70372238A4; Wed, 9 Feb 2022 00:43:34 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 2190hYNP083077; Wed, 9 Feb 2022 00:43:34 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 2190hY7V083076; Wed, 9 Feb 2022 00:43:34 GMT (envelope-from git) Date: Wed, 9 Feb 2022 00:43:34 GMT Message-Id: <202202090043.2190hY7V083076@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Konstantin Belousov Subject: git: 1617ca2c16a3 - stable/13 - vm_object: restore handling of shadow_count for all type of objects List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-branches@freebsd.org X-BeenThere: dev-commits-src-branches@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: kib X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: 1617ca2c16a38482a40b44982e8c2bc4ab6c447c Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1644367415; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=FrkJnph1Yqo5f4Os5hmUchvhTPoP6oF5+YkyGjx+304=; b=JWE+vQOaMbdTw+RhrSTSLeGjVyqRY6qxGU26xwX5tY+h94k1RqgKztbsNlfQw4GWJFDqYn dSST878LuwMcsj4cLR8WMUfeK2m1y85yav/YN1Ehdi1KvCZTb2vIZrhJB85rxV02697Rv7 5vjVC9TkJDlCJSVqlasME0NJln+wxBxXAn7DCGz+U6iovSwfUvF3Gj65MPymkj1uls2p/H SPkHWJD2oPYpOMuvBRIbERi03Cbdq7MDmbb9COFGsqy4PyhmSDBENAoFPXC6miydJMso3G GtkHi9nE6WxNxTX8l1cp37jI/M4eAnmSBi9GEKqeFtuOqPHwkspCRjbQO+8Y7A== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1644367415; a=rsa-sha256; cv=none; b=Z6Bmwh3PdBieHOKkyLl1gReoNSvcKvokbGBUqKbTa/wWifNxcf6jB6V6wpPvAGoiSE6tbQ DlrN5xqyi1Nx8PssUZRdzKrO/Xr4oFeMmxZTegG44kBfIxt0CPUjm9Wd7FwNclGeixELaf dbRT1deBi2YmLhFEmIc87O5RbTlS4hfR3ad7Z/ojCo2BEIcfjaRMxBCLfOhO20haedD84u idWBLYXMT0y4PReZPv80WLdraC1xtYxDjokDVeuM7tSk48ClKrq98wN7tE0e2x2Awh14g7 sYFOHz/+fducFALZVf/RsOp3L83ePEu5duMtoC35N2CVszHU/crt368SywDHVA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch stable/13 has been updated by kib: URL: https://cgit.FreeBSD.org/src/commit/?id=1617ca2c16a38482a40b44982e8c2bc4ab6c447c commit 1617ca2c16a38482a40b44982e8c2bc4ab6c447c Author: Konstantin Belousov AuthorDate: 2021-12-17 20:55:45 +0000 Commit: Konstantin Belousov CommitDate: 2022-02-09 00:42:44 +0000 vm_object: restore handling of shadow_count for all type of objects (cherry picked from commit 4cf9f5d807a7767a2ec6113c99bbba024d08b489) --- sys/vm/vm_object.c | 57 +++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 39 insertions(+), 18 deletions(-) diff --git a/sys/vm/vm_object.c b/sys/vm/vm_object.c index 7bc503f7afdf..94f9bed3d9b8 100644 --- a/sys/vm/vm_object.c +++ b/sys/vm/vm_object.c @@ -198,9 +198,9 @@ vm_object_zdtor(void *mem, int size, void *arg) KASSERT(object->resident_page_count == 0, ("object %p resident_page_count = %d", object, object->resident_page_count)); - KASSERT(object->shadow_count == 0, + KASSERT(atomic_load_int(&object->shadow_count) == 0, ("object %p shadow_count = %d", - object, object->shadow_count)); + object, atomic_load_int(&object->shadow_count))); KASSERT(object->type == OBJT_DEAD, ("object %p has non-dead type %d", object, object->type)); @@ -222,7 +222,7 @@ vm_object_zinit(void *mem, int size, int flags) blockcount_init(&object->paging_in_progress); blockcount_init(&object->busy); object->resident_page_count = 0; - object->shadow_count = 0; + atomic_store_int(&object->shadow_count, 0); object->flags = OBJ_DEAD; mtx_lock(&vm_object_list_mtx); @@ -574,9 +574,11 @@ vm_object_deallocate_anon(vm_object_t backing_object) /* Fetch the final shadow. */ object = LIST_FIRST(&backing_object->shadow_head); - KASSERT(object != NULL && backing_object->shadow_count == 1, + KASSERT(object != NULL && + atomic_load_int(&backing_object->shadow_count) == 1, ("vm_object_anon_deallocate: ref_count: %d, shadow_count: %d", - backing_object->ref_count, backing_object->shadow_count)); + backing_object->ref_count, + atomic_load_int(&backing_object->shadow_count))); KASSERT((object->flags & OBJ_ANON) != 0, ("invalid shadow object %p", object)); @@ -660,7 +662,7 @@ vm_object_deallocate(vm_object_t object) */ if (!refcount_release(&object->ref_count)) { if (object->ref_count > 1 || - object->shadow_count == 0) { + atomic_load_int(&object->shadow_count) == 0) { if ((object->flags & OBJ_ANON) != 0 && object->ref_count == 1) vm_object_set_flag(object, @@ -719,6 +721,14 @@ vm_object_destroy(vm_object_t object) uma_zfree(obj_zone, object); } +static void +vm_object_sub_shadow(vm_object_t object) +{ + KASSERT(object->shadow_count >= 1, + ("object %p sub_shadow count zero", object)); + atomic_subtract_int(&object->shadow_count, 1); +} + static void vm_object_backing_remove_locked(vm_object_t object) { @@ -731,9 +741,9 @@ vm_object_backing_remove_locked(vm_object_t object) KASSERT((object->flags & OBJ_COLLAPSING) == 0, ("vm_object_backing_remove: Removing collapsing object.")); + vm_object_sub_shadow(backing_object); if ((object->flags & OBJ_SHADOWLIST) != 0) { LIST_REMOVE(object, shadow_list); - backing_object->shadow_count--; object->flags &= ~OBJ_SHADOWLIST; } object->backing_object = NULL; @@ -746,13 +756,15 @@ vm_object_backing_remove(vm_object_t object) VM_OBJECT_ASSERT_WLOCKED(object); + backing_object = object->backing_object; if ((object->flags & OBJ_SHADOWLIST) != 0) { - backing_object = object->backing_object; VM_OBJECT_WLOCK(backing_object); vm_object_backing_remove_locked(object); VM_OBJECT_WUNLOCK(backing_object); - } else + } else { object->backing_object = NULL; + vm_object_sub_shadow(backing_object); + } } static void @@ -761,11 +773,11 @@ vm_object_backing_insert_locked(vm_object_t object, vm_object_t backing_object) VM_OBJECT_ASSERT_WLOCKED(object); + atomic_add_int(&backing_object->shadow_count, 1); if ((backing_object->flags & OBJ_ANON) != 0) { VM_OBJECT_ASSERT_WLOCKED(backing_object); LIST_INSERT_HEAD(&backing_object->shadow_head, object, shadow_list); - backing_object->shadow_count++; object->flags |= OBJ_SHADOWLIST; } object->backing_object = backing_object; @@ -781,8 +793,10 @@ vm_object_backing_insert(vm_object_t object, vm_object_t backing_object) VM_OBJECT_WLOCK(backing_object); vm_object_backing_insert_locked(object, backing_object); VM_OBJECT_WUNLOCK(backing_object); - } else + } else { object->backing_object = backing_object; + atomic_add_int(&backing_object->shadow_count, 1); + } } /* @@ -805,6 +819,7 @@ vm_object_backing_insert_ref(vm_object_t object, vm_object_t backing_object) VM_OBJECT_WUNLOCK(backing_object); } else { vm_object_reference(backing_object); + atomic_add_int(&backing_object->shadow_count, 1); object->backing_object = backing_object; } } @@ -831,6 +846,11 @@ vm_object_backing_transfer(vm_object_t object, vm_object_t backing_object) vm_object_backing_insert_locked(object, new_backing_object); VM_OBJECT_WUNLOCK(new_backing_object); } else { + /* + * shadow_count for new_backing_object is left + * unchanged, its reference provided by backing_object + * is replaced by object. + */ object->backing_object = new_backing_object; backing_object->backing_object = NULL; } @@ -1924,9 +1944,9 @@ vm_object_collapse(vm_object_t object) return; KASSERT(object->ref_count > 0 && - object->ref_count > object->shadow_count, + object->ref_count > atomic_load_int(&object->shadow_count), ("collapse with invalid ref %d or shadow %d count.", - object->ref_count, object->shadow_count)); + object->ref_count, atomic_load_int(&object->shadow_count))); KASSERT((backing_object->flags & (OBJ_COLLAPSING | OBJ_DEAD)) == 0, ("vm_object_collapse: Backing object already collapsing.")); @@ -1940,9 +1960,10 @@ vm_object_collapse(vm_object_t object) * all the resident pages in the entire backing object. */ if (backing_object->ref_count == 1) { - KASSERT(backing_object->shadow_count == 1, + KASSERT(atomic_load_int(&backing_object->shadow_count) + == 1, ("vm_object_collapse: shadow_count: %d", - backing_object->shadow_count)); + atomic_load_int(&backing_object->shadow_count))); vm_object_pip_add(object, 1); vm_object_set_flag(object, OBJ_COLLAPSING); vm_object_pip_add(backing_object, 1); @@ -2510,7 +2531,7 @@ bool vm_object_is_active(vm_object_t obj) { - return (obj->ref_count > obj->shadow_count); + return (obj->ref_count > atomic_load_int(&obj->shadow_count)); } static int @@ -2565,7 +2586,7 @@ vm_object_list_handler(struct sysctl_req *req, bool swap_only) kvo->kvo_size = ptoa(obj->size); kvo->kvo_resident = obj->resident_page_count; kvo->kvo_ref_count = obj->ref_count; - kvo->kvo_shadow_count = obj->shadow_count; + kvo->kvo_shadow_count = atomic_load_int(&obj->shadow_count); kvo->kvo_memattr = obj->memattr; kvo->kvo_active = 0; kvo->kvo_inactive = 0; @@ -2779,7 +2800,7 @@ DB_SHOW_COMMAND(object, vm_object_print_static) object->resident_page_count, object->ref_count, object->flags, object->cred ? object->cred->cr_ruid : -1, (uintmax_t)object->charge); db_iprintf(" sref=%d, backing_object(%d)=(%p)+0x%jx\n", - object->shadow_count, + atomic_load_int(&object->shadow_count), object->backing_object ? object->backing_object->ref_count : 0, object->backing_object, (uintmax_t)object->backing_object_offset);