Fresh 7.0 Install: Fatal Trap 12 panic when put under load

[Michael Grant]

On Thu, Sep 11, 2008 at 11:20 AM, Jeremy Chadwick <koitsu at freebsd.org> wrote:
> On Thu, Sep 11, 2008 at 10:38:36AM +0200, Michael Grant wrote:
>> My box crashed again:
>>
>> panic: kmem_malloc(4096): kmem_map too small: 1073741824 total allocated
>> cpuid = 0
>> Uptime: 33d11h12m58s
>> Dumping 3327 MB (2 chunks)
>>   chunk 0: 1MB (151 pages) ... ok
>>   chunk 1: 3327MB (851568 pages)  <---hung here
>>
>> Still no valid dump.
>>
>> There is 4gig of physical memory in the machine.
>>
>> In /boot/loader.conf, I currently have the following:
>>
>> vm.kmem_size=1G
>> vm.kmem_size_max=1G
>> vm.kmem_size_scale=2
>>
>> and in my kernel conf file I have:
>>
>> options         KVA_PAGES=512
>>
>> It stayed up for 33 days this time.  Is there anything else I can do?
>
> First and foremost: are you using ZFS on this machine?  If so, there are
> many tunables you can apply to try and limit this; I'm willing to bet
> it's ARC which is doing it.  See below.
>
> In general, it appears that you need to increase the maximum range of
> kmem.  The kernel attempted to utilise more than 1GB, and your limit is
> 1G.  My machines running RELENG_7 on amd64, with only 2GB of RAM
> installed, use the following tunables in loader.conf:
>
> vm.kmem_size="1536M"
> vm.kmem_size_max="1536M"
>
> If ZFS is in use, I recommend these as well:
>
> vfs.zfs.arc_min="16M"
> vfs.zfs.arc_max="64M"
> vfs.zfs.prefetch_disable="1"
>
> Do not increase kmem_size any larger than 1.5GB; the amount of RAM you
> have in the machine, with regards to RELENG_7, will not help.  This is a
> known limitation which has been fixed in HEAD/CURRENT (where the limit
> has been increased to 512GB).  See the "Kernel" section below; you'll
> see the applicable item.
>
> http://wiki.freebsd.org/JeremyChadwick/Commonly_reported_issues
>
> Your only solution may be to run HEAD/CURRENT.

I am not running ZFS.  My file systems are ufs.

This feels like some sort of memory leak in the kernel.  Giving it
more and more memory just seems to delay the crash.  Are you saying
the crash is fixed in HEAD/CURRENT?

I'm running 6.3 by the way.

I have put your changes into my loader.conf, we'll see how long it
goes this time.  I'm not qute in position to update everything to 7.x
at the moment.

Michael Grant