kmem map too small panic after updating to STABLE-7 r192996

[Kip Macy]

As I mentioned in the initial e-mail, auto-tuning is only safe to rely
on on amd64.

-Kip

On Thu, Jun 4, 2009 at 7:48 AM, Tim Chase <tim at chase2k.com> wrote:
> Hello,
>
> I decided to give the new zfs code a try and upgraded my stable-7 system
> and discovered a panic that I can reproduce at will.
>
> This is an i386 system with 1.5GB of RAM and a single zfs pool:
>
>        appbuild# zpool status
>          pool: backup
>         state: ONLINE
>        status: The pool is formatted using an older on-disk format.  The
> pool can
>                still be used, but some features are unavailable.
>        action: Upgrade the pool using 'zpool upgrade'.  Once this is done,
> the
>                pool will no longer be accessible on older software versions.
>         scrub: none requested
>        config:
>
>                NAME        STATE     READ WRITE CKSUM
>                backup      ONLINE       0     0     0
>                  mirror    ONLINE       0     0     0
>                    ad4     ONLINE       0     0     0
>                    ad6     ONLINE       0     0     0
>
>        errors: No known data errors
>
> I had previously used the following zfs tuning:
>
>        vm.kmem_size="512M"
>        vm.kmem_size_max="512M"
>        vfs.zfs.arc_max="100M"
>
> After getting this panic, I removed these tunables.  The panic happens
> in either case.
>
> Reproducing the panic is easy: I keep a copy of the FreeBSD svn tree
> in a compressed zfs file system (compression ratio runs around 1.35x).
> An "svn update" (or the requisite "svn cleanup" following a system crash)
> in my checkout area will _always_ result in a "kmem map too small" panic.
> Here's a stack trace from my most recent panic:
>
> (kgdb) bt
> #0  doadump () at pcpu.h:196
> #1  0xc052c963 in boot (howto=260) at
> /root/stable-7/sys/kern/kern_shutdown.c:418
> #2  0xc052cb9d in panic (fmt=Variable "fmt" is not available.
> ) at /root/stable-7/sys/kern/kern_shutdown.c:574
> #3  0xc069fa2a in kmem_malloc (map=0xc105408c, size=131072, flags=2) at
> /root/stable-7/sys/vm/vm_kern.c:305
> #4  0xc0696587 in page_alloc (zone=0x0, bytes=131072, pflag=0xe6dbcb47
> "\002", wait=2)
>    at /root/stable-7/sys/vm/uma_core.c:952
> #5  0xc0699000 in uma_large_malloc (size=131072, wait=2) at
> /root/stable-7/sys/vm/uma_core.c:2706
> #6  0xc051af38 in malloc (size=131072, mtp=0xc094f060, flags=2) at
> /root/stable-7/sys/kern/kern_malloc.c:393
> #7  0xc085db00 in zfs_kmem_alloc (size=131072, kmflags=2)
>    at
> /root/stable-7/sys/modules/zfs/../../cddl/compat/opensolaris/kern/opensolaris_kmem.c:74
> #8  0xc08d1c79 in zio_buf_alloc (size=131072)
>    at
> /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:207
> #9  0xc08bcc40 in vdev_queue_io_to_issue (vq=0xc492b334,
> pending_limit=Variable "pending_limit" is not available.
> )
>    at
> /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:227
> #10 0xc08bce42 in vdev_queue_io_done (zio=0xd420a000)
>    at
> /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:313
> #11 0xc08d4162 in zio_vdev_io_done (zio=0xd420a000)
>    at
> /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1847
> #12 0xc08d2532 in zio_execute (zio=0xd420a000)
>    at
> /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:998
> #13 0xc0862aa0 in taskq_thread (arg=0xc44a10cc)
>    at
> /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/os/taskq.c:854
> #14 0xc0506816 in fork_exit (callout=0xc0862960 <taskq_thread>,
> arg=0xc44a10cc, frame=0xe6dbcd38)
>    at /root/stable-7/sys/kern/kern_fork.c:811
> #15 0xc06d4670 in fork_trampoline () at
> /root/stable-7/sys/i386/i386/exception.s:264
> (kgdb) print panicstr
> $1 = 0xc0792320 "kmem_malloc(131072): kmem_map too small: 325656576 total
> allocated"
>
>
> The arc size is now auto-tuned as:
>
>        kstat.zfs.misc.arcstats.c_min: 26214400
>        kstat.zfs.misc.arcstats.c_max: 209715200
>
> and while monitoring kstat.zfs.misc.arcstats.size I noticed it fluctuated
> between around 140 and 150 million.  Interestingly enough, immediately
> before the last panic, It (arcstats.size) had just been reduced from
> 150038268 to 128790020.
>
> I'm going to continue to poke around in my core dumps and see if I can
> figure out what's going on.  Any hints as to what to look for or monitor
> while the system is running would be appreciated.
>
>        Thanks,
>        Tim
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
>



-- 
When bad men combine, the good must associate; else they will fall one
by one, an unpitied sacrifice in a contemptible struggle.

    Edmund Burke