Repeatable ZFS "kmem map too small" panic on 8.0-STABLE

[Artem Belevich]

On Thu, Jan 21, 2010 at 12:46 PM, Gary Corcoran <gcorcoran at rcn.com> wrote:
> Adam McDougall wrote:
>>
>> Put this in /boot/loader.conf:
>> vm.kmem_size="20G"
>>
>> It is intentionally higher than your amount of ram.
>
> Would you mind explaining...
> 1) why this fixes the kmem_map too small problem ?

Because it explicitly makes kmem_map larger.

> 2) why it should be larger than the amount of RAM, and by how much ?

ZFS needs access to a lot of memory for ARC and it allocates/frees
memory fairly randomly. That raises two issues.

First issue is that kernel is by default fairly conservative about its
memory needs. vm.kmem_size which limits address space for in-kernel
memory allocations is by default set to a fairly low value which works
reasonably well in most of the cases. However, for ZFS it needs to be
bumped up allow large amounts of memory to be allocated by ZFS.

Second problem is memory fragmentation. If you set vm.kmem_size ==
physical memory size, over time you may end up with a situation when
there is plenty of physical memory available, but there is no single
contiguous block of address space to map that memory into. FreeBSD
allocator is pretty good about avoiding fragmentation but you still do
need more address space than the amount of memory that could
potentially be allocated. I'd say that vm.kmem_size should be few
multiples of amounts of memory that you're planning to allocate.

Just my $0.02

--Artem