ARC "pressured out", how to control/stabilize ? (reformatted to text/plain)
Vladimir Sharun
atz at ukr.net
Thu Jan 30 12:18:58 UTC 2014
Dear Andriy and FreeBSD community,
L2ARC temporarily turned off by setting secondarycache=none everywhere it was enabled,
so no more leak for one particular day.
Here's the top header:
last pid: 89916; load averages: 2.49, 2.91, 2.89 up 5+19:21:42 14:09:12
561 processes: 2 running, 559 sleeping
CPU: 5.7% user, 0.0% nice, 14.0% system, 1.0% interrupt, 79.3% idle
Mem: 23G Active, 1017M Inact, 98G Wired, 1294M Cache, 3285M Buf, 1997M Free
ARC: 69G Total, 3498M MFU, 59G MRU, 53M Anon, 1651M Header, 4696M Other
Swap:
Here's the calculated vmstat -z (mean all of the allocations, which exceeds 100*1024^2 printed):
UMA Slabs: 199,915M
VM OBJECT: 207,354M
32: 205,558M
64: 901,122M
128: 215,211M
256: 242,262M
4096: 2316,01M
range_seg_cache: 205,396M
zio_buf_512: 1103,31M
zio_buf_16384: 15697,9M
zio_data_buf_16384: 348,297M
zio_data_buf_24576: 129,352M
zio_data_buf_32768: 104,375M
zio_data_buf_36864: 163,371M
zio_data_buf_53248: 100,496M
zio_data_buf_57344: 105,93M
zio_data_buf_65536: 101,75M
zio_data_buf_73728: 111,938M
zio_data_buf_90112: 104,414M
zio_data_buf_106496: 100,242M
zio_data_buf_131072: 61652,5M
dnode_t: 3203,98M
dmu_buf_impl_t: 797,695M
arc_buf_hdr_t: 1498,76M
arc_buf_t: 105,802M
zfs_znode_cache: 352,61M
zio_data_buf_131072 (61652M) + zio_buf_16384 (15698M) = 77350M
easily exceeds ARC total (70G)
Here's the same calculations from exact the same system where L2 was disabled before reboot:
last pid: 63407; load averages: 2.35, 2.71, 2.73 up 8+19:42:54 14:17:33
527 processes: 1 running, 526 sleeping
CPU: 4.8% user, 0.0% nice, 6.6% system, 1.1% interrupt, 87.4% idle
Mem: 21G Active, 1460M Inact, 99G Wired, 1748M Cache, 3308M Buf, 952M Free
ARC: 87G Total, 4046M MFU, 76G MRU, 37M Anon, 2026M Header, 4991M Other
Swap:
and the vmstat -z filtered:
UMA Slabs: 208,004M
VM OBJECT: 207,392M
32: 172,831M
64: 752,226M
128: 210,024M
256: 244,204M
4096: 2249,02M
range_seg_cache: 245,711M
zio_buf_512: 1145,25M
zio_buf_16384: 15170,1M
zio_data_buf_16384: 422,766M
zio_data_buf_20480: 120,742M
zio_data_buf_24576: 148,641M
zio_data_buf_28672: 112,848M
zio_data_buf_32768: 117,375M
zio_data_buf_36864: 185,379M
zio_data_buf_45056: 103,168M
zio_data_buf_53248: 105,32M
zio_data_buf_57344: 122,828M
zio_data_buf_65536: 109,25M
zio_data_buf_69632: 100,406M
zio_data_buf_73728: 126,844M
zio_data_buf_77824: 101,086M
zio_data_buf_81920: 100,391M
zio_data_buf_86016: 101,391M
zio_data_buf_90112: 112,836M
zio_data_buf_98304: 100,688M
zio_data_buf_102400: 106,543M
zio_data_buf_106496: 108,875M
zio_data_buf_131072: 63190,5M
dnode_t: 3437,36M
dmu_buf_impl_t: 840,62M
arc_buf_hdr_t: 1870,88M
arc_buf_t: 114,942M
zfs_znode_cache: 353,055M
Everything seems within ARC total range.
We will try patch attached within few days and will come back with the result.
Thank you for your help.
> on 28/01/2014 11:28 Vladimir Sharun said the following:
> > Dear Andriy and FreeBSD community,
> >
> > After applying this path one of the systems runs fine (disk subsystem load low to moderate
> > - 10-20% busy sustained),
> >
> > Then I saw this patch was merged to the HEAD and we apply it to the one of the systems
> > with moderate to high disk load: 30-60% busy (11.0-CURRENT #7 r261118: Fri Jan 24 17:25:08 EET 2014)
> >
> > Within 4 days we experiencing the same leak(?) as without patch:
> >
> > last pid: 53841; load averages: 4.47, 4.18, 3.78 up 3+16:37:09 11:24:39
> > 543 processes: 6 running, 537 sleeping
> > CPU: 8.7% user, 0.0% nice, 14.6% system, 1.4% interrupt, 75.3% idle
> > Mem: 22G Active, 1045M Inact, 98G Wired, 1288M Cache, 3284M Buf, 2246M Free
> > ARC: 73G Total, 3763M MFU, 62G MRU, 56M Anon, 1887M Header, 4969M Other
> > Swap:
> >
> > The ARC is populated within 30mins under load to the max (90Gb) then start decreasing.
> >
> > The delta between Wiread and ARC total start growing from typical 10-12Gb without L2 enabled
> > to the 25Gb with L2 enabled and counting (4 hours ago was 22Gb delta).
>
> First, have you checked that vmstat -z output contains the same anomaly as for
> in your original report?
>
> If yes, the please try to reproduce the problem with the following debugging patch:
> http://people.freebsd.org/~avg/l2arc-b_tmp_cdata-diag.patch
> Please make sure to compile your kernel (and modules) with INVARIANTS.
>
> --
> Andriy Gapon
> _______________________________________________
> freebsd-current at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe at freebsd.org"
More information about the freebsd-current
mailing list