Repeatable ZFS "kmem map too small" panic on 8.0-STABLE

[Adam McDougall]

On 01/21/10 17:21, Artem Belevich wrote:
> On Thu, Jan 21, 2010 at 12:46 PM, Gary Corcoran<gcorcoran at rcn.com>  wrote:
>> Adam McDougall wrote:
>>>
>>> Put this in /boot/loader.conf:
>>> vm.kmem_size="20G"
>>>
>>> It is intentionally higher than your amount of ram.
>>
>> Would you mind explaining...
>> 1) why this fixes the kmem_map too small problem ?
>
> Because it explicitly makes kmem_map larger.
>
>> 2) why it should be larger than the amount of RAM, and by how much ?
>
> ZFS needs access to a lot of memory for ARC and it allocates/frees
> memory fairly randomly. That raises two issues.
>
> First issue is that kernel is by default fairly conservative about its
> memory needs. vm.kmem_size which limits address space for in-kernel
> memory allocations is by default set to a fairly low value which works
> reasonably well in most of the cases. However, for ZFS it needs to be
> bumped up allow large amounts of memory to be allocated by ZFS.
>
> Second problem is memory fragmentation. If you set vm.kmem_size ==
> physical memory size, over time you may end up with a situation when
> there is plenty of physical memory available, but there is no single
> contiguous block of address space to map that memory into. FreeBSD
> allocator is pretty good about avoiding fragmentation but you still do
> need more address space than the amount of memory that could
> potentially be allocated. I'd say that vm.kmem_size should be few
> multiples of amounts of memory that you're planning to allocate.
>
> Just my $0.02
>
> --Artem
>

Exactly what I would have said, thanks :) I'd imagine the kmem_size 
could be much larger still, closer to kmem_size_max, but I just picked 
20G as a default for my servers that have 8G or less and I haven't seen 
an out of kmem panic for as long as I could raise kmem_size sufficiently 
high (a change was made around 6 months ago).  kmem_size doesn't seem to 
"grow" (much?) towards kmem_size_max, it is what it is, and you need to 
make sure it is big enough for your needs.  I have systems with just one 
gig and they run fine.