From nobody Mon Jan 24 22:13:05 2022 X-Original-To: bugs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id B1309197978A for ; Mon, 24 Jan 2022 22:13:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JjPPY2m0kz3j7p for ; Mon, 24 Jan 2022 22:13:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 1E4CC1DC4B for ; Mon, 24 Jan 2022 22:13:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 20OMD5wb041777 for ; Mon, 24 Jan 2022 22:13:05 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 20OMD5El041776 for bugs@FreeBSD.org; Mon, 24 Jan 2022 22:13:05 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 261448] ZFS VOP_RECLAIM causes long stalls during package builds Date: Mon, 24 Jan 2022 22:13:05 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: markj@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Bug reports List-Archive: https://lists.freebsd.org/archives/freebsd-bugs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-bugs@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1643062385; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=/jez8QSaqEYvbYNUb5z+WvTlojn5UlRSr3eHHS5KASA=; b=veaWZeulQhiKA440hOCWwmxrszeGy8telKZiEoQmT2LMtuW4kqY8AId2Rcw0YGB2BtCWna YG/cCJWQqwvsgcekxAS2FnJEXbfGB4p/SoeE6+eo0fKi5IjNbGS+EkcOlnlumz1gQeFXk0 62ZX/cT8SO7ufLZQF1YgRaWQXCa4Ro9c4go6DVst+FGK+RCE0Rl827JmK/tJhjbV7uRR5v Lga4fFobxqdrKYhTAzfhTUBigC6LoPYCfDZmbIifvc1YJcvbG6BvySvaptFbR54WgKhgQ7 eZxq9qaxbSPB6Grmh3uZAVyYAxygEbwlEbn1S4erYV5Jw6hFaAlKTY3ZyNWE+g== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1643062385; a=rsa-sha256; cv=none; b=BMoWQHBNVIXWiQ8iG612V1WWFiUsrxfba9DOD1Tm1dt9kps3CYgmlkEdcD23NXiKR+38Rd ie/8VV2bG5t319zmEEeAtQU4UrQjO9X2n/VDyDbV4EBO0HXVBPbca8u9JV2MMUXBd5Wvjq lRnTRvkpuSG1dVanMZtcoEKU9fR/yjHhyJy/8HaLWHCgi5AOSZD/CpWRq86x3+rGhYezh3 h7Dwk/BE0SLMFuZqStFP5zZIpBwthqsOKKxdTtK8bLaSMJt6FXFOUCiuRTDxNE3OZdEbJk 9tYR2zFdYRG9qm4Vi7lhmTOn1fJwJ5PX0rX9sGe96eOVxuwrvcQqZiTrSEyRHg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D261448 Bug ID: 261448 Summary: ZFS VOP_RECLAIM causes long stalls during package builds Product: Base System Version: CURRENT Hardware: Any OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: markj@FreeBSD.org I noticed that a system running "poudriere bulk -a" often appears to stall = for several seconds. During a stall, all CPUs are nearly idle even though the system is ostensibly building NCPU packages in parallel. Digging a bit, the problem happens when a package build finishes and poudriere does a "zfs rollback" of some dataset. The kernel implementation of zfs rollback calls zfsvfs_teardown(), which us= es the per-dataset "teardown lock" to suspend all VOPs on the dataset while it walks the list of vnodes associated with the mount point (multiple times) to invalidate the name cache and cached data. Any user threads performing operations on files in the dataset are blocked for this period. The second part of the problem is getnewvnode(), which may call vn_alloc_ha= rd() to reclaim a used vnode if the total number of vnodes in the system is above the "desiredvnodes" threshold. vn_alloc_hard() performs direct reclamation from the vnode free list, and as a part of this recycling it must call VOP_RECLAIM. ZFS' VOP_RECLAIM implementation, like all VOPs, acquires the teardown lock, so it can block for a long time. If we get unlucky and a ba= tch of vnodes belonging to a suspended dataset are at the head of the free list, then the system very quickly gets stuck until the rollback completes. Logging CPU usage during poudriere runs shows lots of idle CPU caused by th= is problem. The amount varies depending on how quickly package builds are finishing. Logging output from "vmstat 1" overnight gives: $ cat vmstat-log | awk '/^[0-9]/{user +=3D $(NF - 2); sys +=3D $(NF - 1); i= dle +=3D $(NF)} END{printf "user %d sys %d idle %d total %d\n", user, sys, idle, use= r + sys + idle}' user 2123240 sys 259657 idle 126830 total 2509727 so something like 5% of CPU is lost. Data collected over a shorter time pe= riod during which there were lots of small package builds shows it can be much worse: user 112162 sys 25396 idle 47896 total 185454 --=20 You are receiving this mail because: You are the assignee for the bug.=