kern.geom.journal.stats.low_mem refers to what?
LukeD at pobox.com
Thu Mar 31 02:07:26 UTC 2011
> I've recently set up gjournal on top of gmirror on FreeBSD 8.2. I understand
> that this setup has a lot of redundant writing. It is working, but I'm not
> sure I've set it up as efficiently as I should.
> During prolonged writes, such as copying large files to the file system
> across the network or producing very large logfiles, the "low_mem" and
> "skipped_bytes" statisics rise rapidly and the system becomes less
> responsive. "top" always reports free memory, so I don't think that's the
> issue. "journal_full" and "wait_for_copy" have never occurred. Here's a
> sample of what happens after a couple hours of intense writing...
> kern.geom.journal.stats.low_mem: 5379
> kern.geom.journal.stats.journal_full: 0
> kern.geom.journal.stats.wait_for_copy: 0
> kern.geom.journal.stats.switches: 7543
> kern.geom.journal.stats.combined_ios: 265318
> kern.geom.journal.stats.skipped_bytes: 935712768
> "low_mem" sounds like a bad thing. What could I do to remedy that? Did I
> make the journal too small? The stats say that "journal_full" has never
> happened, so maybe not? Is there a setting I should tweak?
> The Handbook says 1GB is good enough most of the time, but it also says that
> 3x the amount of physical memory is a good size as well. I compromised
> between the two and made an 8GB journal for this system that has ~4GB of
Replying to myself for posterity... I think I figured it out.
I believe "low_mem" means its running out of gjournal cache, defined by
kern.geom.journal.cache.limit, which is initially set by
kern.geom.journal.cache.divisor, which is initially 2, which means half of
the kernel memory, which is vm.kmem_size, which is 330MB on my i386 8.2
system by default. I don't know where this default comes from, but it
seems common. This gives me about 167MB of cache for journalling.
During big file transfers, I can watch the "journal_data" statistic in
"vmstat -m". It climbs close to the 167MB cache size. That's when I see
kern.geom.journal.stats.low_mem tick over and the stutter happens. So I
think I understand what I'm seeing now.
I could try increasing vm.kmem_size and possibly the KMEM_SIZE option
that's compiled into my kernel as described in the ZFS tuning documents,
since it's a similar issue. This would allow the kernel to have more
memory and thus allow gjournal to use more memory for caching. If I'm
consistently filling the cache faster than I can empty it, making the
cache bigger probably won't help. This journal is on top of a mirror, so
I'm always expecting it to read faster than it writes... Copying large
files locally might always max out the buffer? No harm in trying, I
Or I could just live with it. It's doing what I told it to do.
More information about the freebsd-questions