Tuning kern.maxswzone is minor compared to hangs in "kmem a" state

[Matthew Rezny]

On Sun, 2 Feb 2014 15:59:57 -0800
Adrian Chadd <adrian at freebsd.org> wrote:

> [snip]
> 
> So next time this happens, run "procstat -kka" - this will dump out
> the processes and a basic function call trace for each of them.
> 
> I'll see if there's a way to teach procstat to output line numbers if
> the kernel debug image is available, but generally that's enough to
> then pass to kgdb to figure out the line number.
> 
> 
> -a

I'm not sure if I would be able to run procstat given that top or
reboot fail to run once the kernel is up against the limit.
Fortunately, I haven't had a chance to even try that. I found the
solution shortly after my last message but it took some time to verify.

The solution is to increase vm.kmem_size to give the kernel some room
to grow. That is one of the first things in tuning ZFS, which I had
just started dealing with on i386 on a pair of less obsolete boxes. It
struck me that maybe I should look into this parameter, which is one I
never had occasion to have to touch before. On the box with 384MB RAM
this was defaulting to 120.5MB, but on the box with 256MB RAM it was
defaulting to only 80MB. It appears the default is simply 1/3 available
RAM at boot. Presumably there is some lower bound, however that lower
bound is no longer sufficient with all else default.

I set vm.kmem_size="120M" in loader.conf and after rebooting I saw an
immediate world of difference. I could do svnlite status and it
completed. I put the box through the paces, svn up, buildworld and
kernel, installworld and kernel (so now running 10-STABLE), svn up
again, buildworld -DNO_CLEAN (still takes half a day, but that's far
less than 2 days), installworld again. In other words, it was back to
how it had been while running 9-STABLE. I am perfectly happy to give
more than half the system memory to the kernel to have it stable.

While that box was on the second run through building world, I decided
to collect some numbers. I setup a temporary test VM in VirtualBox (on
a much faster machine) configured with no harddrive image and boot from
ISO. I used both 9.2 and 10.0 release ISOs for i386. I went through
configurations with 128, 192, 256, and 384MB of RAM. For each OS and
RAM combination, I booted the system and collected the entire output of
sysctl. I did not do an stress testing, just collected default values.
Comparing the data what I found is that while many other values
changed, the value of vm.kmem_size stays the same from 9.2 to 10.0 at
any given system memory size. I observed that the lower bound (if any)
takes effect well below my smallest box since the 1/3 trend continued
down all the way down to the smallest test VM with only half the RAM.

It seems clear that with default settings on 10.0, vm.kmem_size is
undersized for low-memory machines even running UFS. It seems common
knowledge this needs to go up for ZFS, but it's complete news to me it
needs to go up for UFS. I have not dug deep enough to determine if
there is a singular culprit or multiple subsystems have grown more
memory hungry over time. From the one panic I got it appears at least
UFS with softupdates can be at least one. The extreme sluggishness if
not complete hang or outright failure of disk I/O is a symptom
consistent with the hypothesis that it has been UFS hitting an
allocation limit in the kernel each time.

Not only is it news to me that this might need to be increased on a UFS
system, but the whole existence of vm.kmem_size is news to me since I
never had to touch it in the past. On amd64, I see it is simply set to
physical RAM, which makes sense, the kernel can grow to fill RAM but
not beyond (don't want to swap it). Obviously it can't simply be equal
to physical RAM on i386, it can't be over 1GB unless KVA_PAGES is
increased. I don't understand why it isn't simply set to min(phy_mem,
1GB) by default. I can understand having a tunable to limit the kernel
from growing for cases where the administrator knows best for some
very specific working set. As the general case, I expect the kernel to
handle maintaining the optimal balance between itself and user programs
and I expect the ideal balance to vary between different workloads.
Having a hard limit on kernel size only serves to limit the kernel's
ability to tune the system according to current conditions.

Is there any benefit to limiting kmem_size on i386? Is there any reason
I should not simply set this vm.kmem_size=min(phys_mem, 1GB) on all my
i386 boxes? For what reason is it stated that KVA_PAGES needs to be
increased when setting kmem_size > 512MB when the default for KVA_PAGES
gives a 1GB kernel memory space?