kern/53416: kmem_map too small after around 12 hours. (5.1-REL)

[Peter Losher]

>Number:         53416
>Category:       kern
>Synopsis:       kmem_map too small after around 12 hours. (5.1-REL)
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Jun 17 12:00:30 PDT 2003
>Closed-Date:
>Last-Modified:
>Originator:     Peter Losher
>Release:        FreeBSD 5.1-RELEASE i386
>Organization:
ISC
>Environment:
System: FreeBSD sf1.isc.org 5.1-RELEASE FreeBSD 5.1-RELEASE #2: Tue Jun 17 05:12:24 UTC 2003 plosher at sf1.isc.org:/usr/src/sys/i386/compile/ISC_FBSD_51_SMP i386

>Description:

So, I recently put a Quad-Xeon server (w/ 4GB of RAM) into production as
primarily as a master FTP server for ISC as well as for FreeBSD (US/IPv6
side of ftp.freebsd.org).  All has been going well for the past couple of
weeks until 5.1 was released. Traffic spiked, and the server started
panicking every 12 hours (give or take 30 min):

> panic: kmem_malloc(4096): kmem_map too small: 377487360 total allocated

So after looking at the archives, I read that FreeBSD had issues with
installed memory above 2GB, so I set the sysctl kern.maxvnodes=130000, but
to no effect:

> panic: kmem_malloc(16384): kmem_map too small: 293519360 total allocated

I have since removed my NMBCLUSTER setting in the kernel, so it would be
set automatically, (also resetting the kern.maxvnodes sysctl back to default) 
I have even set VM_KMEM_SIZE_SCALE="2" (I haven't messed with the other 
KMEM kernel options, as I was hoping to avoid it)  And 13 hours ago I 
updated the kernel with the latest kern_malloc.c file in the RELENG_5_1 branch.

That too also panicked after 12.5 hours this morning, looking at 'netstat -m'
just before it crashed, it had wired memory reserves of 16M, and the amount 
(percentage being used) in the last hour rose steadily from 68% to about 86% 
before the panic occurs.

>How-To-Repeat:
	
>Fix:

The work around is to reboot the box every 11.5 hours, so the 
kmem_map gets reset.  However that is not a viable solution for a 
production box. :(  Is there any way to check and make sure there is not a
leakage, and if not, make sure that FreeBSD is calculating a large enough
kmem_map?

>Release-Note:
>Audit-Trail:
>Unformatted: