6.3-RELEASE-p3 recurring panics on multiple SM PDSMi+

[Kris Kennaway]

Royce Williams wrote:
> Kris Kennaway wrote, on 7/22/2008 12:12 PM:
>> Royce Williams wrote:
>>
>>> db> trace
>>> Tracing pid 71182 tid 100325 td 0xcc08b180
>>> kdb_enter(c095f294) at kdb_enter+0x2b
>>> panic(c09768ad,1000,14000000,c145bc88,1000,...) at panic+0x127
>>> kmem_malloc(c14680c0,1000,102,eba6a8cc,c07e3fa5,...) at kmem_malloc+0x89
>> You forgot to include the panic
> 
> Is there is a way to get the panic after dropping into the debugger?
> This serial console setup has no scrollback, so I couldn't see the
> preceding text.

You can either "show msgbuf", or "x/x panicstr" and then "x/s 0x...." 
where that is the hex value from the previous step.  The latter only 
diplays the format string and not the arguments, but on i386 you can 
read them off from the panic() line in the stack trace.  Actually on 
i386 the panicstr is the first argument (0xc09768ad here).

>> but this is probably the "kmem_map too small" panic.  
> 
> Ah, I see this now, at faq/book.html#KMEM-MAP-TOO-SMALL:
> 
> "Compile your own kernel, and add the VM_KMEM_SIZE_MAX to your kernel
> configuration file, increasing the maximum size to 400 MB (options
> VM_KMEM_SIZE_MAX=419430400). 400 MB appears to be sufficient for
> machines with up to 6 GB of memory."
> 
> 
>> It says that your kernel ran out of memory, and the
>> solution is to fix that situation by giving more memory to the kernel.
>> Increase the vm.kmem_size tunable until your system stops running out of
>> memory on your workload.
> 
> Comparing the FAQ, kern_malloc.c and your mentioning it as tunable,
> please clarify: is the Right Thing to do to use vm.kmem_size, or
> vm.kmem_size_max?

kmem_size_max is used for automatically tuning based on RAM size.  To 
increase the actual value explicitly you just need to tune vm.kmem_size.

> I tried vm.kmem_size_max, which yields:
> 
> # sysctl -a | grep kmem
> vm.kmem_size: 419430400
> vm.kmem_size_max: 419430400
> vm.kmem_size_scale: 3
> 
> 
> Should I contribute some additional language to the FAQ, saying that
> the vm.kmem_size[_max] tunable can be used without recompiling the kernel?

Yes, that would be fantastic!  I would also note that the loader tunable 
is usually more convenient since it doesn't require a kernel recompile, 
and probably reword the claim about 400MB: the memory needed depends 
very much on the workload you are giving your kernel, so the best advice 
is to increase the value until you determine empirically how much you 
need (i.e. the memory exhaustion stops).

You can also use "400M" notation for loader tunables which is often more 
convenient.

Thanks,
Kris

Kris