vm.kmem_size settings doesn't affect loader?

Jeremy Chadwick koitsu at FreeBSD.org
Thu Sep 25 15:44:02 UTC 2008

On Thu, Sep 25, 2008 at 04:10:07PM +0100, Pete French wrote:
> > These are the tuning settings I use:
> >
> > vm.kmem_size="1536M"
> > vm.kmem_size_max="1536M"
> > vfs.zfs.arc_min="16M"
> > vfs.zfs.arc_max="64M"
> >
> > The entire copying process took almost 2 hours.  Not once did I
> > experience kmem exhaustion.  I can *guarantee* that I would have crashed
> > the box numerous times had I not tuned the machine with the values
> > above.
> I am interested in the last two values you have there - I also use ZFS and
> tune to prevent memory exhaustion. I tune kmem_size, but I have not bothered
> toi set the arc_min - and the default appears to be 16M anyway. The arc_max
> value I do set, but I got dreadful performance when I had it at 6M, so
> I upped it to 128M.

ARC tuning is what's necessary to induce ZFS memory constraints.  If you
only set kmem_size/max, you will likely run into cases where the ARC
grows to such a size that kmem exhaustion happens.  Even the Solaris ZFS
"do not tune ZFS" article outlines this fact (read it all, including the


Just keep in mind this is specific to Solaris, but the problems
conceptually are identical to FreeBSD.

> This has not resulted in any panics, and gives much better
> performance. As I understand it there is only one ARC, and it comes from
> kernel space too, so setting it to an extra 64 meg should not use that much
> more memory, yes ?

The ARC uses kmem.  "Should not use that much more memory" is a matter
of opinion; if an additional 64MB given to ARC causes kmem exhaustion
(there's no global standard for this; keep reading, I explain it near
the bottom of my mail), then decreasing the ARC max is necesary.

On the bright side, it's very easy to determine how much memory is being
dedicated to the ARC.  sysctl kstat.zfs.misc will disclose this data
and I believe the following statistics are the ones you have to examine
for this issue:


On my system (the one using the tuning parameters you're questioning):

kstat.zfs.misc.arcstats.p: 60487680
kstat.zfs.misc.arcstats.c: 67108864
kstat.zfs.misc.arcstats.c_min: 16777216
kstat.zfs.misc.arcstats.c_max: 67108864
kstat.zfs.misc.arcstats.size: 67092480

For those wanting to read about the ARC and what it's for:


> My setup is:
> vm.kmem_size="1024M"
> vm.kmem_size_max="1024M"
> vfs.zfs.arc_max="128M"
> vfs.zfs.prefetch_disable=1
> This is on a 2gb amd64 machine, and works beautifully, with no out of
> memory errors and panics. I have been trying to panic this for a few
> days. I will probably chnage the kmem size to 1536 which is what I use on
> other machines though.

Disabling prefetch shouldn't have any effect on stability -- many of us
(including FreeBSD core members) have found that disabling prefetch
increases system responsiveness during heavy I/O, and results in a less
"bursty" system.  I disable prefetch, but it should have no bearing on
the issue we're discussing.

I'm sure the memory tuning values work fine for you, but they do not for
me; I can absolutely panic the machine with a kmem size of 1GB.  How?
Because it's entirely dependent upon the type of ZFS pools/setup a
person has, how much actual load they put on ZFS, and probably a hundred
other things.

Let's adhere to the K.I.S.S. concept, and not turn this thread into
a "tuning micro-management" thread.

What end-users want is ZFS that "just works", which on Solaris 10 it
does.  But I have no qualms with tuning ZFS on FreeBSD, though others
seem to.  A "tuning guide" cannot easily be written because there are
too many factors to take into consideration.  Besides, such a guide
(documenting "how it all interconnects, memory-wise") would end up being
tailored to the technical, which won't help end-users.  The existing
"ZFS tuning guide" for FreeBSD keeps things vague for a good reason --
I'm absolutely 100% positive an extensive document would just confuse
the living daylights out of people.

| Jeremy Chadwick                                jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |

More information about the freebsd-stable mailing list