ARC "pressured out", how to control/stabilize ? (reformatted to text/plain)

[Vladimir Sharun]

Dear Andriy and FreeBSD community,

After applying this path one of the systems runs fine (disk subsystem load low to moderate 
- 10-20% busy sustained),

Then I saw this patch was merged to the HEAD and we apply it to the one of the systems 
with moderate to high disk load: 30-60% busy (11.0-CURRENT #7 r261118: Fri Jan 24 17:25:08 EET 2014)

Within 4 days we experiencing the same leak(?) as without patch: 

last pid: 53841;  load averages:  4.47,  4.18,  3.78     up 3+16:37:09  11:24:39
543 processes: 6 running, 537 sleeping
CPU:  8.7% user,  0.0% nice, 14.6% system,  1.4% interrupt, 75.3% idle
Mem: 22G Active, 1045M Inact, 98G Wired, 1288M Cache, 3284M Buf, 2246M Free
ARC: 73G Total, 3763M MFU, 62G MRU, 56M Anon, 1887M Header, 4969M Other
Swap:

The ARC is populated within 30mins under load to the max (90Gb) then start decreasing.

The delta between Wiread and ARC total start growing from typical 10-12Gb without L2 enabled
to the 25Gb with L2 enabled and counting (4 hours ago was 22Gb delta).

L2ARC statistics:

L2 ARC Size: (Adaptive)                         291.63  GiB
        Header Size:                    0.25%   734.14  MiB

L2 ARC Evicts:
        Lock Retries:                           682
        Upon Reading:                           0

L2 ARC Breakdown:                               106.56m
        Hit Ratio:                      29.04%  30.95m
        Miss Ratio:                     70.96%  75.62m
        Feeds:                                  317.18k

So, again, what shall we do to better understand/mitigate the problem further ?

Thank you.

> on 15/01/2014 12:28 Vitalij Satanivskij said the following:
> > Dear Andriy and FreeBSD community,
> > 
> > Andriy Gapon wrote:
> > AG> on 14/01/2014 07:27 Vladimir Sharun said the following:
> > AG> > Dear Andriy and FreeBSD community,
> > AG> > 
> > AG> >> I am not sure if the buffers are leaked somehow or if they are actually in use.
> > AG> >> It's one of the very few places where data buffers are allocated without
> > AG> >> charging ARC. In all other places it's quite easy to match allocations and
> > AG> >> deallocations. But in L2ARC it is not obvious that all buffers get freed or
> > AG> >> when that happens.
> > AG> > 
> > AG> > After one week under load I think we figure out the cause: it's L2ARC. 
> > AG> > Here's the top's header for 7d17h of the runtime:
> > AG> > 
> > AG> > last pid: 46409; load averages: 0.37, 0.62, 0.70 up 7+17:14:01 07:24:10
> > AG> > 173 processes: 1 running, 171 sleeping, 1 zombie
> > AG> > CPU: 2.0% user, 0.0% nice, 3.5% system, 0.4% interrupt, 94.2% idle
> > AG> > Mem: 8714M Active, 14G Inact, 96G Wired, 1929M Cache, 3309M Buf, 3542M Free
> > AG> > ARC: 85G Total, 2558M MFU, 77G MRU, 28M Anon, 1446M Header, 4802M Other
> > AG> > 
> > AG> > ARC related tunables:
> > AG> > 
> > AG> > vm.kmem_size="110G"
> > AG> > vfs.zfs.arc_max="90G"
> > AG> > vfs.zfs.arc_min="42G"
> > AG> > 
> > AG> > For more than 7 days of hard runtime the picture clearly shows: 
> > AG> > Wired minus ARC = 11..12Gb, ARC grow and shrinks in 80-87Gb range and the
> > AG> > system runs just fine.
> > AG> > 
> > AG> > So what shall we do with L2ARC leakage ?
> > AG> 
> > AG> 
> > AG> Could you please try this patch
> > AG> http://cr.illumos.org/~webrev/skiselkov/3995/illumos-gate.patch ?
> > AG> 
> > 
> > While applying path to curent version of arc.c (r260622) I'm found next truble with compilation 
> > 
> > olaris/uts/common/fs/zfs/arc.c -o arc.o
> > /usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:4628:18: error: use of
> > undeclared identifier 'abl2'
> > trim_map_free(abl2->b_dev->l2ad_vdev, abl2->b_daddr,
> > ^
> > 1 error generated.
> > *** Error code 1
> > 
> > 
> > the code is - 
> > 
> > if (zio->io_error != 0) {
> > /*
> > * Error - drop L2ARC entry.
> > */
> > list_remove(buflist, ab);
> > ARCSTAT_INCR(arcstat_l2_asize, -l2hdr->b_asize);
> > ab->b_l2hdr = NULL;
> > trim_map_free(abl2->b_dev->l2ad_vdev, abl2->b_daddr,
> > ab->b_size, 0);
> > kmem_free(l2hdr, sizeof (l2arc_buf_hdr_t));
> > ARCSTAT_INCR(arcstat_l2_size, -ab->b_size);
> > }
> > 
> > 
> > Looks like it's part is freebsd specific changes.
> > Can somebody help with this part of code ?
> > 
> 
> The first hunk of the patch is renaming of abl2 to l2hdr.
> 
> -- 
> Andriy Gapon
> _______________________________________________
> freebsd-current at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe at freebsd.org"
>