[Bug 293737] BIND 9.20 causes runaway RSS with non-matching memory statistics
Date: Wed, 11 Mar 2026 15:14:55 UTC
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=293737
Bug ID: 293737
Summary: BIND 9.20 causes runaway RSS with non-matching memory
statistics
Product: Base System
Version: 15.0-STABLE
Hardware: Any
OS: Any
Status: New
Severity: Affects Only Me
Priority: ---
Component: kern
Assignee: bugs@FreeBSD.org
Reporter: ondrej@sury.org
Attachment #268720 text/plain
mime type:
Created attachment 268720
--> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=268720&action=edit
Script to parse and compare two jemalloc stats print output in JSON
Hi,
this is as much bug report as it is plea for help. I don't know yet :).
We've got multiple (2 external and 1 internal) people running FreeBSD (both
14.x and 15.x) where BIND 9.18 is fine, but BIND 9.20 has runaway RSS memory
problem.
I wrote this simple patch that prints out the jemalloc memory statistics once
in a while:
```patch
diff --git a/bin/named/server.c b/bin/named/server.c
index 25eb888e7ad..9edd24f374a 100644
--- a/bin/named/server.c
+++ b/bin/named/server.c
@@ -6725,10 +6725,26 @@ pps_timer_tick(void *arg) {
UNUSED(arg);
+#ifdef JEMALLOC_API_SUPPORTED
+ /*
+ * Purge all jemalloc arenas before dumping stats so that RSS
+ * reflects actual usage rather than retained-but-free pages.
+ */
+ {
+ char purge_mctl[64];
+ snprintf(purge_mctl, sizeof(purge_mctl), "arena.%u.purge",
+ MALLCTL_ARENAS_ALL);
+ (void)mallctl(purge_mctl, NULL, NULL, NULL, 0);
+ }
+
+ malloc_stats_print(NULL, NULL, "J");
+
+#endif /* JEMALLOC_API_SUPPORTED */
+
/*
* Don't worry about wrapping as the overflow result will be right.
*/
- dns_pps = (requests - oldrequests) / 1200;
+ dns_pps = (requests - oldrequests) / 6000;
oldrequests = requests;
}
@@ -8321,7 +8337,7 @@ apply_configuration(cfg_obj_t *effectiveconfig, cfg_obj_t
*bindkeys,
&interval);
}
- isc_interval_set(&interval, 1200, 0);
+ isc_interval_set(&interval, 6000, 0);
isc_timer_start(server->pps_timer, isc_timertype_ticker, &interval);
isc_interval_set(&interval, named_g_tat_interval, 0);
```
and asked LLM to code parser for the JSON (sorry, I am overloaded) - attached.
Then the user who reported the issue ran this on his production system and
basically, the output was like this (TL;DR, full output below):
```
Global counters:
Allocated : 44.9 MiB (application-usable bytes)
Active : 57.1 MiB (pages backing allocations)
Metadata : 7.43 MiB (jemalloc bookkeeping)
Resident : 64.2 MiB (RSS contribution)
Mapped : 91.1 MiB (total mmap'd)
Retained : 45.9 MiB (returned to OS, still mapped)
```
At the same time their RSS was:
```
SIZE = 495 MB
RES = 412 MB
```
My running theory is that this is an artifact of jemalloc using `MADV_FREE`
which doesn't immediately return the memory to the kernel, but they claim:
> When it happened the first time it did not crash, but I saw it using swap space, which was crazy. So there was pressure. I will check again.
I have a similar internal user running resolver on FreeBSD with following
results:
> my home production resolver has an RSS of 328M, but stats channel only reports 34M malloced
I have so many questions :), but if I reduce them to just couple:
- is this something that might happen (MADV_DONTUSE and MADV_FREE difference
between Linux and FreeBSD)
- is there any memory region that would manifest in RSS/RES, but not in
jemalloc memory statistics (kernel memory?)
- is there a reliable way how to test this? (rctl?)
- is there a way how to enable jemalloc profiling? (although thinking about
this now, this might not help as jemalloc shows 1/10 of the RES).
Full jemalloc JSON summary...
```
==============================================================================
jemalloc statistics: primero.json
==============================================================================
Version : 5.2.1-0-gea6b3e973b477b8061e0076bb257dbd7f3faa756
Configured arenas: 16
Dirty decay : 10000 ms
Muzzy decay : 0 ms
Global counters:
Allocated : 44.7 MiB (application-usable bytes)
Active : 55.0 MiB (pages backing allocations)
Metadata : 7.34 MiB (jemalloc bookkeeping)
Resident : 61.9 MiB (RSS contribution)
Mapped : 89.0 MiB (total mmap'd)
Retained : 41.0 MiB (returned to OS, still mapped)
Fragmentation / overhead:
Internal frag : 10.2 MiB (22.8% of allocated)
External frag : 6.99 MiB (12.7% of active)
Metadata ratio : 11.9% (metadata / resident)
Map overhead : 27.0 MiB (mapped − resident)
Allocation classes (merged):
Small : 34.7 MiB (77.6%) live= 178,308 total_req= 524,094,282
Large : 10.0 MiB (22.4%) live= 90 total_req= 135,715
Metadata breakdown (merged):
Base (arena metadata) : 6.96 MiB
Internal (extent meta): 394 KiB
Tcache bytes : 1.62 MiB
Purge activity (merged):
Dirty: purges= 26762 madvise calls= 50499 pages purged= 315777
Muzzy: purges= 0 madvise calls= 0 pages purged= 0
Per-arena summary:
Arena Threads Active Resident Mapped Retained Tcache
Uptime
───── ─────── ────────── ────────── ────────── ────────── ──────────
────────
0 1 23.8 MiB 29.7 MiB 31.8 MiB 9.18 MiB 255 KiB
1.7h
1 1 0.00 B 40.0 KiB 2.00 MiB 0.00 B 60.3 KiB
1.7h
2 1 9.68 MiB 9.88 MiB 11.7 MiB 5.32 MiB 242 KiB
1.7h
3 1 9.36 MiB 9.55 MiB 11.4 MiB 5.64 MiB 225 KiB
1.7h
4 1 9.45 MiB 9.65 MiB 11.4 MiB 5.55 MiB 231 KiB
1.7h
5 1 280 KiB 324 KiB 2.27 MiB 1.73 MiB 18.5 KiB
1.7h
6 1 280 KiB 324 KiB 2.27 MiB 1.73 MiB 15.5 KiB
1.7h
7 1 276 KiB 324 KiB 2.27 MiB 1.73 MiB 19.7 KiB
1.7h
8 1 276 KiB 324 KiB 2.27 MiB 1.73 MiB 17.9 KiB
1.7h
9 1 368 KiB 420 KiB 2.36 MiB 1.64 MiB 124 KiB
1.7h
10 1 332 KiB 384 KiB 2.32 MiB 1.68 MiB 126 KiB
1.7h
11 1 332 KiB 384 KiB 2.32 MiB 1.68 MiB 126 KiB
1.7h
12 1 424 KiB 476 KiB 2.41 MiB 1.59 MiB 127 KiB
1.7h
13 1 140 KiB 184 KiB 2.14 MiB 1.86 MiB 71.3 KiB
1.7h
Top 15 active small-object bins (merged):
Size CurRegs CurBytes Slabs SlabBytes Util Requests
──────── ───────── ────────── ────── ────────── ────── ────────────
640 B 10,612 6.48 MiB 392 7.66 MiB 84.6% 21,717,960
192 B 26,784 4.90 MiB 479 5.61 MiB 87.4% 8,087,561
128 B 21,725 2.65 MiB 801 3.13 MiB 84.8% 4,789,890
160 B 15,803 2.41 MiB 149 2.91 MiB 82.9% 43,602,308
256 B 9,674 2.36 MiB 771 3.01 MiB 78.4% 19,252,466
768 B 2,745 2.01 MiB 239 2.80 MiB 71.8% 28,845,970
112 B 18,403 1.97 MiB 86 2.35 MiB 83.6% 61,918,340
7.00 KiB 270 1.85 MiB 105 2.87 MiB 64.3% 6,604
448 B 3,079 1.32 MiB 56 1.53 MiB 85.9% 2,422,481
5.00 KiB 236 1.15 MiB 75 1.46 MiB 78.7% 5,150,995
320 B 3,680 1.12 MiB 82 1.60 MiB 70.1% 24,067,799
224 B 4,835 1.03 MiB 54 1.48 MiB 70.0% 11,736,777
1.50 KiB 656 984 KiB 145 1.70 MiB 56.6% 10,664,478
2.00 KiB 468 936 KiB 270 1.05 MiB 86.7% 102,880
1.25 KiB 669 836 KiB 54 1.05 MiB 77.4% 1,484,655
Top 10 active large extent classes (merged):
Size Count Total
────────── ────── ────────────
1.25 MiB 4 5.00 MiB
48.0 KiB 23 1.08 MiB
96.0 KiB 10 960 KiB
224 KiB 2 448 KiB
28.0 KiB 14 392 KiB
24.0 KiB 16 384 KiB
192 KiB 2 384 KiB
384 KiB 1 384 KiB
80.0 KiB 4 320 KiB
320 KiB 1 320 KiB
==============================================================================
jemalloc statistics: segundo.json
==============================================================================
Version : 5.2.1-0-gea6b3e973b477b8061e0076bb257dbd7f3faa756
Configured arenas: 16
Dirty decay : 10000 ms
Muzzy decay : 0 ms
Global counters:
Allocated : 44.9 MiB (application-usable bytes)
Active : 57.1 MiB (pages backing allocations)
Metadata : 7.43 MiB (jemalloc bookkeeping)
Resident : 64.2 MiB (RSS contribution)
Mapped : 91.1 MiB (total mmap'd)
Retained : 45.9 MiB (returned to OS, still mapped)
Fragmentation / overhead:
Internal frag : 12.2 MiB (27.0% of allocated)
External frag : 7.07 MiB (12.4% of active)
Metadata ratio : 11.6% (metadata / resident)
Map overhead : 26.9 MiB (mapped − resident)
Allocation classes (merged):
Small : 34.6 MiB (77.1%) live= 177,786
total_req=2,590,735,106
Large : 10.3 MiB (22.9%) live= 92 total_req= 697,764
Metadata breakdown (merged):
Base (arena metadata) : 7.04 MiB
Internal (extent meta): 394 KiB
Tcache bytes : 1.46 MiB
Purge activity (merged):
Dirty: purges=134784 madvise calls=250914 pages purged= 1618620
Muzzy: purges= 0 madvise calls= 0 pages purged= 0
Per-arena summary:
Arena Threads Active Resident Mapped Retained Tcache
Uptime
───── ─────── ────────── ────────── ────────── ────────── ──────────
────────
0 1 24.8 MiB 30.8 MiB 32.8 MiB 8.21 MiB 124 KiB
10.0h
1 1 0.00 B 44.0 KiB 2.00 MiB 2.00 MiB 58.2 KiB
10.0h
2 1 10.2 MiB 10.4 MiB 12.2 MiB 9.83 MiB 247 KiB
10.0h
3 1 9.62 MiB 9.83 MiB 11.6 MiB 5.38 MiB 217 KiB
10.0h
4 1 9.81 MiB 10.0 MiB 11.8 MiB 5.19 MiB 193 KiB
10.0h
5 1 280 KiB 324 KiB 2.27 MiB 1.73 MiB 18.3 KiB
10.0h
6 1 272 KiB 320 KiB 2.27 MiB 1.73 MiB 24.2 KiB
10.0h
7 1 292 KiB 340 KiB 2.29 MiB 1.71 MiB 19.1 KiB
10.0h
8 1 288 KiB 336 KiB 2.28 MiB 1.72 MiB 17.9 KiB
10.0h
9 1 368 KiB 420 KiB 2.36 MiB 1.64 MiB 123 KiB
10.0h
10 1 424 KiB 480 KiB 2.41 MiB 1.59 MiB 126 KiB
10.0h
11 1 332 KiB 384 KiB 2.32 MiB 1.68 MiB 126 KiB
10.0h
12 1 384 KiB 436 KiB 2.38 MiB 1.62 MiB 126 KiB
10.0h
13 1 140 KiB 184 KiB 2.14 MiB 1.86 MiB 71.3 KiB
10.0h
Top 15 active small-object bins (merged):
Size CurRegs CurBytes Slabs SlabBytes Util Requests
──────── ───────── ────────── ────── ────────── ────── ────────────
640 B 10,598 6.47 MiB 395 7.71 MiB 83.8% 108,408,887
192 B 26,897 4.92 MiB 488 5.72 MiB 86.1% 39,843,287
128 B 21,453 2.62 MiB 815 3.18 MiB 82.3% 23,242,382
256 B 10,625 2.59 MiB 784 3.06 MiB 84.7% 94,180,364
160 B 15,708 2.40 MiB 161 3.14 MiB 76.2% 216,237,523
112 B 18,248 1.95 MiB 88 2.41 MiB 81.0% 306,090,074
7.00 KiB 274 1.87 MiB 108 2.95 MiB 63.4% 8,907
768 B 2,439 1.79 MiB 248 2.91 MiB 61.5% 143,796,124
448 B 3,081 1.32 MiB 57 1.56 MiB 84.5% 11,783,239
320 B 3,732 1.14 MiB 87 1.70 MiB 67.0% 120,484,335
5.00 KiB 208 1.02 MiB 74 1.45 MiB 70.3% 25,663,705
224 B 4,620 1011 KiB 73 2.00 MiB 49.4% 58,447,087
2.00 KiB 482 964 KiB 283 1.11 MiB 85.2% 484,575
1.50 KiB 626 939 KiB 159 1.86 MiB 49.2% 53,205,906
1.25 KiB 673 841 KiB 55 1.07 MiB 76.5% 7,011,903
Top 10 active large extent classes (merged):
Size Count Total
────────── ────── ────────────
1.25 MiB 4 5.00 MiB
48.0 KiB 23 1.08 MiB
96.0 KiB 10 960 KiB
320 KiB 3 960 KiB
24.0 KiB 17 408 KiB
28.0 KiB 14 392 KiB
384 KiB 1 384 KiB
80.0 KiB 4 320 KiB
128 KiB 2 256 KiB
224 KiB 1 224 KiB
==============================================================================
jemalloc comparison
==============================================================================
[A] primero.json
[B] segundo.json
Version A: 5.2.1-0-gea6b3e973b477b8061e0076bb257dbd7f3faa756
Version B: 5.2.1-0-gea6b3e973b477b8061e0076bb257dbd7f3faa756
Global counters:
[A] [B] Delta
────────────────────── ────────────── ────────────── ────────────────
Allocated 44.7 MiB 44.9 MiB +205 KiB
Active 55.0 MiB 57.1 MiB +2.15 MiB
Metadata 7.34 MiB 7.43 MiB +87.8 KiB
Resident 61.9 MiB 64.2 MiB +2.23 MiB
Mapped 89.0 MiB 91.1 MiB +2.15 MiB
Retained 41.0 MiB 45.9 MiB +4.85 MiB
Fragmentation:
[A] [B] Delta
────────────────────── ────────────── ────────────── ────────────────
Internal frag 10.2 MiB 12.2 MiB +1.95 MiB
External frag 6.99 MiB 7.07 MiB +88.0 KiB
Metadata 7.34 MiB 7.43 MiB +87.8 KiB
Internal frag % 22.8% 27.0%
Allocation classes (merged):
[A] [B] Delta
────────────────────── ────────────── ────────────── ────────────────
Small allocated 34.7 MiB 34.6 MiB -75.2 KiB
Large allocated 10.0 MiB 10.3 MiB +280 KiB
Small live objects 178,308 177,786 -522
Large live objects 90 92 +2
Tcache bytes 1.62 MiB 1.46 MiB -169 KiB
Base metadata 6.96 MiB 7.04 MiB +87.8 KiB
Internal metadata 394 KiB 394 KiB (same)
Purge activity (merged):
[A] [B] Delta
────────────────────── ────────────── ────────────── ────────────────
Dirty purges 26,762 134,784 +108,022
Dirty madvise 50,499 250,914 +200,415
Dirty pages purged 315,777 1,618,620 +1,302,843
Muzzy purges 0 0 (same)
Top 15 small-bin changes by byte delta (merged):
Size Regs[A] Regs[B] Bytes[A] Bytes[B] Delta
Util[A] Util[B]
──────── ───────── ───────── ────────── ────────── ────────────
─────── ───────
256 B 9,674 10,625 2.36 MiB 2.59 MiB + 238 KiB
78.4% 84.7%
768 B 2,745 2,439 2.01 MiB 1.79 MiB -230 KiB
71.8% 61.5%
5.00 KiB 236 208 1.15 MiB 1.02 MiB -140 KiB
78.7% 70.3%
512 B 147 294 73.5 KiB 147 KiB + 73.5 KiB
51.0% 87.5%
224 B 4,835 4,620 1.03 MiB 1011 KiB -47.0 KiB
70.0% 49.4%
1.50 KiB 656 626 984 KiB 939 KiB -45.0 KiB
56.6% 49.2%
1.00 KiB 316 359 316 KiB 359 KiB + 43.0 KiB
69.3% 80.1%
128 B 21,725 21,453 2.65 MiB 2.62 MiB -34.0 KiB
84.8% 82.3%
8.00 KiB 7 11 56.0 KiB 88.0 KiB + 32.0 KiB
100.0% 100.0%
2.00 KiB 468 482 936 KiB 964 KiB + 28.0 KiB
86.7% 85.2%
7.00 KiB 270 274 1.85 MiB 1.87 MiB + 28.0 KiB
64.3% 63.4%
192 B 26,784 26,897 4.90 MiB 4.92 MiB + 21.2 KiB
87.4% 86.1%
112 B 18,403 18,248 1.97 MiB 1.95 MiB -17.0 KiB
83.6% 81.0%
48.0 B 7,600 7,246 356 KiB 340 KiB -16.6 KiB
63.2% 59.0%
320 B 3,680 3,732 1.12 MiB 1.14 MiB + 16.2 KiB
70.1% 67.0%
Top 10 large-extent changes by byte delta (merged):
Size Cnt[A] Cnt[B] Bytes[A] Bytes[B] Delta
────────── ─────── ─────── ──────────── ──────────── ────────────
320 KiB 1 3 320 KiB 960 KiB + 640 KiB
224 KiB 2 1 448 KiB 224 KiB -224 KiB
192 KiB 2 1 384 KiB 192 KiB -192 KiB
32.0 KiB 2 3 64.0 KiB 96.0 KiB + 32.0 KiB
24.0 KiB 16 17 384 KiB 408 KiB + 24.0 KiB
16.0 KiB 9 9 144 KiB 144 KiB 0.00 B
28.0 KiB 14 14 392 KiB 392 KiB 0.00 B
48.0 KiB 23 23 1.08 MiB 1.08 MiB 0.00 B
80.0 KiB 4 4 320 KiB 320 KiB 0.00 B
96.0 KiB 10 10 960 KiB 960 KiB 0.00 B
Per-arena resident memory:
Arena Resident[A] Resident[B] Delta Thr[A] Thr[B]
───── ──────────── ──────────── ──────────── ────── ──────
0 29.7 MiB 30.8 MiB +1.01 MiB 1 1
1 40.0 KiB 44.0 KiB +4.00 KiB 1 1
2 9.88 MiB 10.4 MiB +512 KiB 1 1
3 9.55 MiB 9.83 MiB +284 KiB 1 1
4 9.65 MiB 10.0 MiB +372 KiB 1 1
5 324 KiB 324 KiB (same) 1 1
6 324 KiB 320 KiB -4.00 KiB 1 1
7 324 KiB 340 KiB +16.0 KiB 1 1
8 324 KiB 336 KiB +12.0 KiB 1 1
9 420 KiB 420 KiB (same) 1 1
10 384 KiB 480 KiB +96.0 KiB 1 1
11 384 KiB 384 KiB (same) 1 1
12 476 KiB 436 KiB -40.0 KiB 1 1
13 184 KiB 184 KiB (same) 1 1
```
--
You are receiving this mail because:
You are the assignee for the bug.