ZFS performance degradation over time

Miroslav Lachman 000.fbsd at quip.cz
Sun Jan 24 11:32:23 UTC 2010


Garrett Moore wrote:
> I've been watching my memory usage and I have no idea what is consuming
> memory as 'Active'.
>
> Last night I had around 6500MB 'Active' again, 1500MB Wired, no inact, ~30MB
> buf, no free, and ~100MB swap used. My performance copying ZFS->ZFS was
> again slow (<1MB/s). I tried killing rTorrent and no significant amount of
> memory was reclaimed - maybe 100MB. `ps aux` showed no processes using any
> significant amount of memory, and I was definitely nowhere near 6500MB
> usage.
>
> I tried running the perl oneliner again to hog a bunch of memory, and almost
> all of the Active memory was IMMEDIATELY marked as Free, and my performance
> was excellent again.
>
> I'm not sure what in userland could be causing the issue. The only things
> I've installed are rTorrent, lighttpd, samba, smartmontools, vim, bash,
> Python, Perl, and SABNZBd. There is nothing that *should* be consuming any
> serious amount of memory.

Last night I tried ZFS with pool on iSCSI connected Dell MD3000i and I 
was suprised by too low speed of simple cp -a command (copying from UFS 
partition to ZFS) The write speed was about 2MB/s only.

After looking in to ARC stuff, I realized some weird values:

ARC Size:
          Current Size:             1 MB (arcsize)
          Target Size (Adaptive):   205 MB (c)
          Min Size (Hard Limit):    205 MB (zfs_arc_min)
          Max Size (Hard Limit):    1647 MB (zfs_arc_max)

(stats from script http://cuddletech.com/arc_summary/
freebsd version 
http://bitbucket.org/koie/arc_summary/changeset/dbe14d2cf52b/ )

I don't know why it shows Current Size 1MB.

I tried the perl oneliner from this thread. Then I got about 5GB free 
merory and Target Size and Current Size growed to 1647MB. Write speed 
increase to about 8MB/s and after few minuts slowly dropped to 2MB/s and 
ARC Current Size dropped to 1MB again.

This server is not in production and was idle. Just copying the data 
from one partition to another.

Today I tried serving the data by Lighttpd.
There is impressive iSCSI read performance - because of ZFS prefetch, it 
can achieve 880Mbits of read from iSCSI, but serving by Lighttpd only 
about 66Mbits

bce0 - internet
bce1 - iSCSI to storage MD3000i

       bce0                bce1
  Kbps in  Kbps out   Kbps in  Kbps out
  2423.22  65481.56  855970.7   4348.73
  2355.26  63911.74  820561.3   4846.08
  2424.87  65998.62  848937.1   4312.37
  2442.78  66544.95  858019.0   4356.64


iostat -x
                         extended device statistics
device     r/s   w/s    kr/s    kw/s wait svc_t  %b
da1      1596.8   3.6 102196.7    22.2   13   7.4  97
da1      1650.2   2.9 105612.7    55.7   16   7.4 103
da1      1647.3   0.0 105422.9     0.0   13   7.2 100
da1      1636.5   2.3 104735.4    20.0   16   7.3 100
da1      1642.9   0.0 105141.1     0.0   13   7.3 100


~/bin/arcstat.pl -f 
Time,read,hits,Hit%,miss,miss%,dmis,dm%,mmis,mm%,arcsz,c 30
     Time  read  hits  Hit%  miss  miss%  dmis  dm%  mmis  mm%  arcsz     c
12:18:05   16K   15K    94   838      5   570    3     1    0   16933296 
   215902720
12:18:36   16K   15K    94   839      5   571    3     0    0   21488288 
   215902720
12:19:06   16K   15K    94   836      5   569    3     1    0   17228688 
   215902720
12:19:37   16K   15K    94   839      5   572    3     4    1   22002672 
   215902720
12:20:07   16K   15K    94   841      5   570    3     1    0   27784960 
   215902720
12:20:38   16K   15K    94   838      5   569    3     0    0   21839472 
   215902720
12:21:08   16K   15K    94   837      5   568    3     0    0   28244992 
   215902720
12:21:39   16K   15K    94   833      5   565    3     1    0   28744416 
   215902720
12:22:09   16K   15K    94   842      5   576    3     4    1   28646656 
   215902720
12:22:39   16K   15K    94   840      5   575    3     3    0   28903696 
   215902720
12:23:10   15K   15K    94   821      5   561    3     0    0   28765904 
   215902720
12:23:40   16K   15K    94   828      5   566    3     0    0   28395840 
   215902720
12:24:11   16K   15K    94   828      5   568    3     0    0   32063408 
   215902720
12:24:41   16K   15K    94   834      5   570    3     0    0   29800976 
   215902720
12:25:12   15K   15K    94   820      5   562    3     1    0   29066512 
   215902720


# ~/bin/arc_summary.pl
System Memory:
          Physical RAM:  8169 MB
          Free Memory :  0 MB

ARC Size:
          Current Size:             22 MB (arcsize)
          Target Size (Adaptive):   205 MB (c)
          Min Size (Hard Limit):    205 MB (zfs_arc_min)
          Max Size (Hard Limit):    1647 MB (zfs_arc_max)

ARC Size Breakdown:
          Most Recently Used Cache Size:           5%    11 MB (p)
          Most Frequently Used Cache Size:        94%    194 MB (c-p)

ARC Efficency:
          Cache Access Total:             81843958
          Cache Hit Ratio:      95%       78525502       [Defined State 
for buffer]
          Cache Miss Ratio:      4%       3318456        [Undefined 
State for Buffer]
          REAL Hit Ratio:       95%       78418994       [MRU/MFU Hits Only]

          Data Demand   Efficiency:    97%
          Data Prefetch Efficiency:    10%

         CACHE HITS BY CACHE LIST:
           Anon:                       --%        Counter Rolled.
           Most Recently Used:          2%        2209869 (mru) 
  [ Return Customer ]
           Most Frequently Used:       97%        76209125 (mfu) 
  [ Frequent Customer ]
           Most Recently Used Ghost:    1%        965711 (mru_ghost) 
  [ Return Customer Evicted, Now Back ]
           Most Frequently Used Ghost:  0%        176871 (mfu_ghost) 
  [ Frequent Customer Evicted, Now Back ]
         CACHE HITS BY DATA TYPE:
           Demand Data:                97%        76770304
           Prefetch Data:               0%        126644
           Demand Metadata:             2%        1628528
           Prefetch Metadata:           0%        26
         CACHE MISSES BY DATA TYPE:
           Demand Data:                63%        2122089
           Prefetch Data:              32%        1063449
           Demand Metadata:             4%        132894
           Prefetch Metadata:           0%        24
---------------------------------------------


# sysctl kstat.zfs.misc.arcstats
kstat.zfs.misc.arcstats.hits: 75409326
kstat.zfs.misc.arcstats.misses: 3144748
kstat.zfs.misc.arcstats.demand_data_hits: 73731356
kstat.zfs.misc.arcstats.demand_data_misses: 2003526
kstat.zfs.misc.arcstats.demand_metadata_hits: 1551917
kstat.zfs.misc.arcstats.demand_metadata_misses: 132730
kstat.zfs.misc.arcstats.prefetch_data_hits: 126027
kstat.zfs.misc.arcstats.prefetch_data_misses: 1008468
kstat.zfs.misc.arcstats.prefetch_metadata_hits: 26
kstat.zfs.misc.arcstats.prefetch_metadata_misses: 24
kstat.zfs.misc.arcstats.mru_hits: 2105758
kstat.zfs.misc.arcstats.mru_ghost_hits: 914887
kstat.zfs.misc.arcstats.mfu_hits: 73197609
kstat.zfs.misc.arcstats.mfu_ghost_hits: 171171
kstat.zfs.misc.arcstats.deleted: 2367973
kstat.zfs.misc.arcstats.recycle_miss: 412788
kstat.zfs.misc.arcstats.mutex_miss: 2865
kstat.zfs.misc.arcstats.evict_skip: 17459
kstat.zfs.misc.arcstats.hash_elements: 2478
kstat.zfs.misc.arcstats.hash_elements_max: 28921
kstat.zfs.misc.arcstats.hash_collisions: 86135
kstat.zfs.misc.arcstats.hash_chains: 25
kstat.zfs.misc.arcstats.hash_chain_max: 3
kstat.zfs.misc.arcstats.p: 14908416
kstat.zfs.misc.arcstats.c: 215902720
kstat.zfs.misc.arcstats.c_min: 215902720
kstat.zfs.misc.arcstats.c_max: 1727221760
kstat.zfs.misc.arcstats.size: 30430560
kstat.zfs.misc.arcstats.hdr_size: 555072
kstat.zfs.misc.arcstats.l2_hits: 0
kstat.zfs.misc.arcstats.l2_misses: 0
kstat.zfs.misc.arcstats.l2_feeds: 0
kstat.zfs.misc.arcstats.l2_rw_clash: 0
kstat.zfs.misc.arcstats.l2_writes_sent: 0
kstat.zfs.misc.arcstats.l2_writes_done: 0
kstat.zfs.misc.arcstats.l2_writes_error: 0
kstat.zfs.misc.arcstats.l2_writes_hdr_miss: 0
kstat.zfs.misc.arcstats.l2_evict_lock_retry: 0
kstat.zfs.misc.arcstats.l2_evict_reading: 0
kstat.zfs.misc.arcstats.l2_free_on_write: 0
kstat.zfs.misc.arcstats.l2_abort_lowmem: 0
kstat.zfs.misc.arcstats.l2_cksum_bad: 0
kstat.zfs.misc.arcstats.l2_io_error: 0
kstat.zfs.misc.arcstats.l2_size: 0
kstat.zfs.misc.arcstats.l2_hdr_size: 0
kstat.zfs.misc.arcstats.memory_throttle_count: 135489


This is on FreeBSD 7.2-STABLE #0: Sun Dec  6 23:21:17 CET 2009 
root at dust.hrej.cz:/usr/obj/usr/src/sys/GENERIC  amd64

Can somebody tell me, why ARC Current Size is dropping too low? (1-20MB 
if arc_min is 205MB)

The system have 8GB of memory and 8 CPU cores:

last pid: 83605;  load averages:  0.17,  0.15,  0.10   up 36+10:34:34 
12:29:05
58 processes:  1 running, 56 sleeping, 1 zombie
CPU:  0.1% user,  0.0% nice,  2.3% system,  1.7% interrupt, 95.8% idle
Mem: 237M Active, 6259M Inact, 1154M Wired, 138M Cache, 827M Buf, 117M Free
Swap: 8192M Total, 96K Used, 8192M Free

I have no loader.conf tunning on this machine.

Miroslav Lachman


More information about the freebsd-stable mailing list