kmem_map too small: 260046848 total allocated

[Marc G. Fournier]

 After about a day and a half or so of uptime, I'm getting the
aforementioned panic on the server ... better then having it hang solid,
but right now I'm not sure if this is replacing it, or just one being
triggered earlier then the other ...

 First scan through Google, I came across some posts talking about
NMBCLUSTERS ... since its at the same settings as my other server (the
default) and it hasn't experienced similar problems, I'm guessing that
that isn't the cause ...

 Now, on neptune (the one with issues), I have KVA_PAGES set to 292,
which, when it crashed, had vm.kvm_free:  104857600 available (I had it at
the default 256, and kvm_free drop'd to 0 within 24hrs, so I raised it a
little bit to give it some buffer space) ... but never raised the
VM_KMEM_SIZE values above their defaults ...

 Is there some tunable that I need to/can increase for this?

 A backtrace of the crash, if that shows anything:

#0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
#1  0xb714e4f3 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:316
#2  0xb714e965 in panic (fmt=0xb72635e0 "kmem_malloc(%ld): kmem_map too small: %ld total allocated") at /usr/src/sys/kern/kern_shutdown.c:595
#3  0xb71fccd7 in kmem_malloc (map=0xb729d100, size=16384, flags=2) at /usr/src/sys/vm/vm_kern.c:329
#4  0xb7149c5f in malloc (size=16384, type=0xb7287c60, flags=2) at /usr/src/sys/kern/kern_malloc.c:198
#5  0xb71ec113 in softdep_disk_io_initiation (bp=0xccc3e848) at /usr/src/sys/ufs/ffs/ffs_softdep.c:2989
#6  0xb718953b in spec_strategy (ap=0xf8237be0) at /usr/src/sys/miscfs/specfs/spec_vnops.c:453
#7  0xb7188fd1 in spec_vnoperate (ap=0xf8237be0) at /usr/src/sys/miscfs/specfs/spec_vnops.c:119
#8  0xb71f7f4d in ufs_vnoperatespec (ap=0xf8237be0) at /usr/src/sys/ufs/ufs/ufs_vnops.c:2394
#9  0xb71f77dd in ufs_strategy (ap=0xf8237c24) at vnode_if.h:944
#10 0xb71f7f1d in ufs_vnoperate (ap=0xf8237c24) at /usr/src/sys/ufs/ufs/ufs_vnops.c:2376
#11 0xb717459b in bwrite (bp=0xccc3e848) at vnode_if.h:944
#12 0xb717a082 in vop_stdbwrite (ap=0xf8237c88) at /usr/src/sys/kern/vfs_default.c:344
#13 0xb7179ea9 in vop_defaultop (ap=0xf8237c88) at /usr/src/sys/kern/vfs_default.c:152
#14 0xb71f7f1d in ufs_vnoperate (ap=0xf8237c88) at /usr/src/sys/ufs/ufs/ufs_vnops.c:2376
#15 0xb7175505 in vfs_bio_awrite (bp=0xccc3e848) at vnode_if.h:1193
#16 0xb71f1bd7 in ffs_fsync (ap=0xf8237cf4) at /usr/src/sys/ufs/ffs/ffs_vnops.c:220
#17 0xb71f051e in ffs_sync (mp=0xc05d2e00, waitfor=2, cred=0xbd042a00, p=0xb72ab3c0) at vnode_if.h:558
#18 0xb717f647 in sync (p=0xb72ab3c0, uap=0x0) at /usr/src/sys/kern/vfs_syscalls.c:577
#19 0xb714e28e in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:235
#20 0xb714e965 in panic (fmt=0xb72635e0 "kmem_malloc(%ld): kmem_map too small: %ld total allocated") at /usr/src/sys/kern/kern_shutdown.c:595
#21 0xb71fccd7 in kmem_malloc (map=0xb729d100, size=4096, flags=0) at /usr/src/sys/vm/vm_kern.c:329
#22 0xb7149c5f in malloc (size=12, type=0xb727bd20, flags=0) at /usr/src/sys/kern/kern_malloc.c:198
#23 0xb718b154 in union_dircache (vp=0xea261f80, p=0xf7ee5ba0) at /usr/src/sys/miscfs/union/union_subr.c:1232
#24 0xb718b320 in union_dircheck (p=0xf7ee5ba0, vp=0xf8237ed4, fp=0xcae50440) at /usr/src/sys/miscfs/union/union_subr.c:1311
#25 0xb7182b2d in getdirentries (p=0xf7ee5ba0, uap=0xf8237f80) at /usr/src/sys/kern/vfs_syscalls.c:3043
#26 0xb723b2e9 in syscall2 (frame={tf_fs = -65489, tf_es = 47, tf_ds = -1228996561, tf_edi = 512, tf_esi = 3584, tf_ebp = -1228979904, tf_isp = -131891244, tf_ebx = 672444300, tf_edx = 141521408, tf_ecx = 1, tf_eax = 196,
      tf_trapno = 683727476, tf_err = 2, tf_eip = 672125312, tf_cs = 31, tf_eflags = 514, tf_esp = -1228980332, tf_ss = 47}) at /usr/src/sys/i386/i386/trap.c:1175
#27 0xb722876b in Xint0x80_syscall ()
(kgdb) up 3
#3  0xb71fccd7 in kmem_malloc (map=0xb729d100, size=16384, flags=2) at /usr/src/sys/vm/vm_kern.c:329
329                             panic("kmem_malloc(%ld): kmem_map too small: %ld total allocated",
(kgdb) list
324                             mb_map_full = TRUE;
325                             printf("Out of mbuf clusters - adjust NMBCLUSTERS or increase maxusers!\n");
326                             return (0);
327                     }
328                     if ((flags & M_NOWAIT) == 0)
329                             panic("kmem_malloc(%ld): kmem_map too small: %ld total allocated",
330                                     (long)size, (long)map->size);
331                     return (0);
332             }
333             offset = addr - VM_MIN_KERNEL_ADDRESS;
(kgdb) print flags
$1 = 2