[Bug 275594] High CPU usage by arc_prune; analysis and fix
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Tue, 13 Feb 2024 07:05:09 UTC
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=275594
--- Comment #51 from Seigo Tanimura <seigo.tanimura@gmail.com> ---
(In reply to Seigo Tanimura from comment #44)
The updated fix is now ready to share.
As of writing this, the test on releng/13.2 is running well without any
unexpected build errors. Hopefully all of the latest fixes are ready for the
review and merge.
* Updates
- Accelerate the ZFS vnode recycling when there are any ARC eviction waiters.
- Recycle the nullfs vnodes, triggered by the vm_lowmem kernel events.
* Github Sources
All of the sources are under
https://github.com/altimeter-130ft/freebsd-freebsd-src.
- Branches and Git Commit Hashes
| | Git
Commit Hash
| Fix Branch |
Base Branch | Fix + Counter Branch | Base
============+=========================================================+=================
main | topic-openzfs-arc_prune-regulation-fix |
57ddfad884
| topic-openzfs-arc_prune-regulation-counters |
------------+---------------------------------------------------------+-----------------
stable/14 | stable/14-topic-openzfs-arc_prune-regulation-fix |
20a6f4779a
| stable/14-topic-openzfs-arc_prune-regulation-counters |
------------+---------------------------------------------------------+-----------------
releng/14.0 | releng/14.0-topic-openzfs-arc_prune-regulation-fix |
4edf3b8073
| releng/14.0-topic-openzfs-arc_prune-regulation-counters |
------------+---------------------------------------------------------+-----------------
stable/13 | stable/13-topic-openzfs-arc_prune-regulation-fix |
9d2f548bbe
| stable/13-topic-openzfs-arc_prune-regulation-counters |
------------+---------------------------------------------------------+-----------------
releng/13.3 | releng/13.3-topic-openzfs-arc_prune-regulation-fix |
24eb518714
| releng/13.3-topic-openzfs-arc_prune-regulation-counters |
------------+---------------------------------------------------------+-----------------
releng/13.2 | releng/13.2-topic-openzfs-arc_prune-regulation-fix |
c78c31d2ef
| releng/13.2-topic-openzfs-arc_prune-regulation-counters |
| Git Commit Hash
| FreeBSD-EN-23:18.openzfs | ZFS & VFS | Nullfs | Counters
Base Branch | Backport | Fix | Fix | (Not for
merging)
============+==========================+============+============+===================
main | N/A | cb4a97bd6b | 7bbbce6313 | ed4932a2b4
------------+--------------------------+------------+------------+-------------------
stable/14 | N/A | f1e1c524a3 | 69f7866c36 | 4e5471ade7
------------+--------------------------+------------+------------+-------------------
releng/14.0 | N/A | af25ebd2d0 | d13f4d74a0 | 09103b5f9c
------------+--------------------------+------------+------------+-------------------
stable/13 | c733dcff2e | 9c65b44d3d | e2bac2ef1e | d52eb31227
------------+--------------------------+------------+------------+-------------------
releng/13.3 | ca178f626f | 90bfaef58d | bd4cceca37 | aad948883e
------------+--------------------------+------------+------------+-------------------
releng/13.2 | c733dcff2e | 9c65b44d3d | e2bac2ef1e | d52eb31227
* poudriere-bulk(8) Results
- Contents of Result Chart Archives
- poudriere-bulk-(base-branch)-(timestamp).7z
- vnodes.png
- The overall vnodes.
- vnode-free-calls.png
- The calls to the vnode recycle functions.
- freed-vnodes.png
- The vnodes freed for recycling.
- nullfs-nodes.png
- The nullfs vnodes.
- vm_lowmem-kernel-events.png
- The vm_lowmem kernel events.
- poudriere-bulk-(base-branch)-(timestamp)-zfs.7z
- zfs-arc-dnode-size.png
- The ZFS ARC dnode size in bytes.
- zfs-arc-pruning-regulation.png
- The ZFS ARC pruning requests and regulation.
- zfs-znodes-and-dnodes.png
- The ZFS ARC znodes and dnodes.
- zfs-dnodes-and-freeing-activity.png
- The ZFS ARC dnodes and pruning activity.
- poudriere-bulk-(base-branch)-(timestamp)-zfs-sizes.7z
- zfs-arc-mru-metadata-size.png
- The ZFS ARC MRU metadata size.
- zfs-arc-mru-data-size.png
- The ZFS ARC MRU data size.
- zfs-arc-mfu-metadata-size.png
- The ZFS ARC MFU metadata size.
- zfs-arc-mfu-data-size.png
- The ZFS ARC MFU data size.
- Some charts and traces are not available in 13.x due to the major version
difference.
- Common Setup
- sysctl(3)
- vfs.zfs.arc_max: 4294967296
- 4GB.
- vfs.zfs.arc.dnode_limit: 0 (default)
- kstat.zfs.misc.arcstats.arc_dnode_limit: 322122547 (calculated
automatically)
- poudriere-bulk(8)
- USE_TMPFS="wrkdir data localbase"
- releng/14.0
- Date
- 09 Feb 2024 00:30Z - 09 Feb 2024 09:07Z
- Build time
- 09:06:22 (234 pkgs / hr)
- Failed port(s): 3
- japanese/xv: Expected, port Makefile problem?
- graphics/gimp-app: Expected, a compiler internal error.
- java/eclipse: Expected, JVM out of heap.
- Skipped port(s): 2
- graphics/gimp: graphics/gimp-app.
- print/gimp-gutenprint: graphics/gimp-app.
- Result Chart Archive
- poudriere-bulk-14_0_release_p4-2024-02-09_09h30m00s.7z, Attachment
#248418
- poudriere-bulk-14_0_release_p4-2024-02-09_09h30m00s-zfs.7z, Attachment
#248419
- poudriere-bulk-14_0_release_p4-2024-02-09_09h30m00s-zfs-sizes.7z,
Attachment #248420
- releng/13.3
- Date
- 08 Feb 2024 03:20Z - 09 Feb 2024 12:01Z
- Build time
- 09:00:46 (237 pkgs / hr)
- Failed port(s): 1
- japanese/xv: Expected, port Makefile problem?
- Skipped port(s): 0
- Result Chart Archive
- poudriere-bulk-13_3_beta_1-2024-02-08_12h20m00s.7z, Attachment #248421
- poudriere-bulk-13_3_beta_1-2024-02-08_12h20m00s-zfs.7z, Attachment
#248422
- poudriere-bulk-13_3_beta_1-2024-02-08_12h20m00s-zfs-sizes.7z, Attachment
#248423
- Missing traces
- The total sizes of each ARC states and types. (Data and metadata)
- releng/13.2
- In progress.
- Started at 13 Feb 2024 02:40Z.
- main, stable/14, stable/13
- No plan for the poudriere-bulk(8) tests for now.
- stable/14 and stable/13 are expected to behave like releng/14.0 and
releng/13.3, respectively.
- Analysis and Findings
- The ZFS ARC dnodes stayed at most ~400MB.
- Before the nullfs fix: monotonic increase in general.
- The ZFS ARC dnodes decreased when only a few builders were running.
- The vnodes stayed at most ~1.2M.
- Before the nullfs fix: up to ~3M.
- The nullfs node number is controlled well as per the build load.
- There are often the ARC eviction waiters in the late stage of
poudriere-bulk(8).
- The tmpfs wrkdirs hold a lot of files for both the build and
prerequisites.
- There are almost always some ARC-prunable vnodes.
- The ZFS ARC always have some evictable parts.
- Almost all data in the ZFS ARC is evictable.
--
You are receiving this mail because:
You are the assignee for the bug.