ARC size limit

[Ronald Klop]

On Tue, 04 Nov 2014 07:29:48 +0100, George Kola <georgekola at gmail.com>  
wrote:

> Hi All,
>        This is my first post to freebsd-stable fresh of Meet BSD  
> California 2014.  We are switching our entire production to FreeBSD. Our  
> storage servers have  256 GB of RAM , 4 TB of SSD and 40 TB of spinning  
> disks. We are running ZFS root and the SSD is configured as L2ARC.  We  
> are running FreeBSD 10.1 RC3
>        I am finding that on all our machines, ARC is somehow limited to  
> < 64 GB of memory and we have a huge inactive memory (180 G). The  
> surprising thing is that ARC seems to have almost the same limit  (< 64  
> GB) on all of our storage boxes and ARC is not growing even though L2ARC  
> hit shows that there is advantage in growing ARC.
>        Any help/pointers is appreciated.
>        What I am trying to do is to tune ZFS for our workload. We are  
> hoping that we get a high hit rate.
>        Thanks to Justin Gibbs and Allan Jude for initial pointers and  
> help. They suggested posting to the mailing list to get further help.
>
>        I have pasted top output and zfs-stats output below and yes UMA  
> is enabled.
>
> Thanks,
> George

What is actually your problem? Do you just want higher numbers of ARC  
usage or is your system slower than expected?
And what is your usage of these 40TB. Are all 40TB accessed a lot or is 64  
GB of ARC equal to the working set of data.

Ronald.



>
>
> top
> last pid: 27458;  load averages:  3.30,  5.42,   
> 5.34                                                                                                 
> up 6+09:59:30  05:38:49
> 71 processes:  1 running, 70 sleeping
> CPU:  4.2% user,  0.0% nice,  4.6% system,  0.2% interrupt, 90.9% idle
> Mem: 11G Active, 181G Inact, 52G Wired, 1368M Cache, 4266M Free
> ARC: 47G Total, 1555M MFU, 41G MRU, 35M Anon, 3984M Header, 709M Other
> Swap: 64G Total, 2874M Used, 61G Free, 4% Inuse
>
>
>
> sysctl vfs.zfs.zio.use_uma
> vfs.zfs.zio.use_uma: 1
>
>
>
>
> zfs-mon -a output
>
> ZFS real-time cache activity monitor
> Seconds elapsed:  62
>
> Cache hits and misses:
>                                   1s    10s    60s    tot
>                      ARC hits:   124    126    103    101
>                    ARC misses:    35     46     29     28
>          ARC demand data hits:    55     90     61     61
>        ARC demand data misses:    20     32     18     17
>      ARC demand metadata hits:    69     36     42     40
>    ARC demand metadata misses:     9     13     10      9
>        ARC prefetch data hits:     0      0      0      0
>      ARC prefetch data misses:     6      1      1      1
>    ARC prefetch metadata hits:     0      0      0      0
>  ARC prefetch metadata misses:     0      0      0      0
>                    L2ARC hits:    16     28     14     14
>                  L2ARC misses:    19     18     15     14
>                   ZFETCH hits:   592   2842   2098   2047
>                 ZFETCH misses:   308   1326    507    494
>
> Cache efficiency percentage:
>                           10s    60s    tot
>                   ARC:  73.26  78.03  78.29
>       ARC demand data:  73.77  77.22  78.21
>   ARC demand metadata:  73.47  80.77  81.63
>     ARC prefetch data:   0.00   0.00   0.00
> ARC prefetch metadata:   0.00   0.00   0.00
>                 L2ARC:  60.87  48.28  50.00
>                ZFETCH:  68.19  80.54  80.56
>
>
>
>
> zfs-stats -a output
>
> ZFS real-time cache activity monitor
> Seconds elapsed:  62
>
> Cache hits and misses:
>                                   1s    10s    60s    tot
>                      ARC hits:   124    126    103    101
>                    ARC misses:    35     46     29     28
>          ARC demand data hits:    55     90     61     61
>        ARC demand data misses:    20     32     18     17
>      ARC demand metadata hits:    69     36     42     40
>    ARC demand metadata misses:     9     13     10      9
>        ARC prefetch data hits:     0      0      0      0
>      ARC prefetch data misses:     6      1      1      1
>    ARC prefetch metadata hits:     0      0      0      0
>  ARC prefetch metadata misses:     0      0      0      0
>                    L2ARC hits:    16     28     14     14
>                  L2ARC misses:    19     18     15     14
>                   ZFETCH hits:   592   2842   2098   2047
>                 ZFETCH misses:   308   1326    507    494
>
> Cache efficiency percentage:
>                           10s    60s    tot
>                   ARC:  73.26  78.03  78.29
>       ARC demand data:  73.77  77.22  78.21
>   ARC demand metadata:  73.47  80.77  81.63
>     ARC prefetch data:   0.00   0.00   0.00
> ARC prefetch metadata:   0.00   0.00   0.00
>                 L2ARC:  60.87  48.28  50.00
>                ZFETCH:  68.19  80.54  80.56
>
>
>
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"