6.3-RELEASE-p3 recurring panics on multiple SM PDSMi+

[Kris Kennaway]

Royce Williams wrote:
> Royce Williams wrote, on 7/22/2008 10:38 PM:
>> Jeremy Chadwick wrote, on 7/22/2008 9:34 PM:
>>> On Tue, Jul 22, 2008 at 11:45:30AM -0800, Royce Williams wrote:
>>>> We have 10 SuperMicro PDSMi+ 5015M-MTs that are panic'ing every few
>>>> days.  This started shortly after upgrade from 6.2-RELEASE to
>>>> 6.3-RELEASE with freebsd-update.
>>> We use the same hardware (board and chassis), and have no such problems
>>> running both RELENG_6 and RELENG_7.
>>>
>>> I don't think your issue is specific to the board or chassis.  Kris's
>>> explanation makes a lot more sense.  :-)
>> Jeremy/Kris/Clifton -
>>
>> Looks like we have consensus. :-)  Thanks, all of you, for your
>> helpful insight.
>>
>> I've bumped vm.kmem_size up to 400M on half of the affected boxes,
>> leaving the other half as a control group.  I'll report back once I
>> have something to report.
> 
> After having bumped up to 400M, a few boxes panic'd again yesterday.  
> I caught a core, and it is "kmem_map too small", just as Kris 
> suspected:
> 
> Jul 31 15:38:05 [redacted] savecore: reboot after panic: kmem_malloc(4096): kmem_map too small: 419430400 total allocated
> 
> 
> The docs state that 400M should be plenty for systems up to 6G, but 
> Kris said earlier in this thread that it's better to say 'increase 
> until the pain stops'. :-)  Accordingly, I have some some follow-up 
> questions; hopefully, this will be useful to others.
> 
> - What is a reasonable increment? (I'm trying 448M next).
> 
> - What are the practical and hard maximums?
> 
> - I suspect that it's worth trying to make kmem 'as big as I need, but 
> no bigger', so that non-kernel memory is also maximized?
> 
> - In a larger sense, if 400M is probably big enough for 6G systems, 
> and these are 4G systems, should I be suspicious that 400M isn't
> cutting it?  In other words, is there a point at which should I be 
> looking for obvious places where the kernel is eating too much memory 
> and reduce them, rather than feeding it more?  
> 
> For example, I recall now that a network guy in my group did some 
> sysctl tuning relating to networking on these systems, and I see 
> from man tuning(7) that a number of these tweaks (obviously) can 
> cause increased kernel consumption.
> 
> $ egrep -v '^#|^$' /etc/sysctl.conf | sort
> kern.corefile=/var/cores/%U/%N-%P.core
> kern.ipc.maxsockbuf=8388608
> kern.ipc.maxsockets=32768
> kern.ipc.nmbclusters=65535
> kern.ipc.somaxconn=4096
> kern.maxfiles=262144
> kern.maxfilesperproc=65535
> net.inet.ip.portrange.first=8192
> net.inet.ip.portrange.hifirst=8192
> net.inet.ip.portrange.hilast=65535
> net.inet.ip.portrange.last=65535
> net.inet.ipf.fr_tcpclosed=60
> net.inet.ipf.fr_tcpclosewait=120
> net.inet.ipf.fr_tcphalfclosed=300
> net.inet.ipf.fr_udptimeout=120
> net.inet.tcp.delayed_ack=0
> net.inet.tcp.inflight.enable=0
> net.inet.tcp.msl=10000
> net.inet.tcp.mssdflt=1460
> net.inet.tcp.recvspace=65536
> net.inet.tcp.rfc1323=1
> net.inet.tcp.sendspace=65536
> net.inet.udp.maxdgram=57344
> net.inet.udp.recvspace=65535
> vfs.nfs.iodmax=32
> vfs.nfs.iodmin=8
> 
> 
> My apologies for not including this sooner.  I didn't think of it
> until yesterday, primarily because it had been fine under 6.2.  In
> retrospect, that was bad reasoning.
> 
> Royce
> 

The statement that "400MB should be enough for up to 6GB" is completely 
bogus.  The amount of memory your kernel needs is a function of the work 
you give it to do.

On i386 the kernel only has 1GB of address space though you can increase 
it by tuning KVA_PAGES at the expense of less memory available to user 
processes (everything comes out of 4GB address space).  So that is the 
upper bound although other things need to fit in there too.

On amd64 the situation is more complicated but on older versions than 
8.0 there is 2GB for the kernel address space and in practise a limit of 
about 1500MB of kmem.

It is possible that you are hitting a memory leak but I would just 
increase kmem further and see if it persists.  Looking at vmstat -m etc 
may help to figure out if something is leaking over time.

Kris