[ZFS] still kmem_too_small on recent current
Dmitry Marakasov
amdmi3 at amdmi3.ru
Fri Jun 5 01:53:27 UTC 2009
Hi!
Just got a kmem_too_small panic on yesterday's current after writing
30GB disk image via NFS.
This is amd64 system, with 8GB mem and those tunables:
vm.kmem_size_max="2G"
vm.kmem_size="2G"
vfs.zfs.arc_max="1G"
ZFS is 6x1TB raidz2.
I have a coredump of it, so just ask if you need additional details.
---
#0 doadump () at pcpu.h:223
#1 0xffffffff8054bec6 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:420
#2 0xffffffff8054c356 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:576
#3 0xffffffff806ed86e in kmem_malloc (map=0xffffff00010000e0, size=131072, flags=2) at /usr/src/sys/vm/vm_kern.c:304
#4 0xffffffff806e5e75 in uma_large_malloc (size=131072, wait=2) at /usr/src/sys/vm/uma_core.c:3001
#5 0xffffffff8053b271 in malloc (size=131072, mtp=0xffffffff80e1a1e0, flags=2) at /usr/src/sys/kern/kern_malloc.c:391
#6 0xffffffff80d90ca6 in vdev_queue_io_to_issue (vq=0xffffff00065bf420, pending_limit=Variable "pending_limit" is not available.
) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:227
#7 0xffffffff80d90e1c in vdev_queue_io_done (zio=0xffffff018e1995a0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:313
#8 0xffffffff80da2370 in zio_vdev_io_done (zio=0xffffff013bc26870) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1845
#9 0xffffffff80da0a10 in zio_execute (zio=0xffffff013bc26870) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:996
#10 0xffffffff80d49f1c in taskq_thread (arg=Variable "arg" is not available.
) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/os/taskq.c:854
#11 0xffffffff80525fad in fork_exit (callout=0xffffffff80d49d48 <taskq_thread>, arg=0xffffff000188fd80, frame=0xffffff80c516cc90) at /usr/src/sys/kern/kern_fork.c:829
#12 0xffffffff8076d56e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:552
---
I've monitored kstat.zfs.misc.arcstats.size, top and vmstat during
it, and here are the observations:
- normal copying process uses ~1GB arc (sysctl) and ~1GB wired (top),
which I assume includes arc.
- actual disk writes are not continuous, ZFS writes data in >512MB
packs.
- usually those writes are pretty uniform (a write in ~10 seconds,
while thoughput is 50MB/s, array can handle 400MB/s writes), but
sometimes the write is either larger, or the disks can't keep up,
or both - those moments are visible as much higher CPU load for
3-5 seconds, wired kicks up to 1700 mb and more, and arc decreases
to ~800MB. I assume the latter is recently committed VM backpressure,
and the former means panic if it rolls over kmem_max.
Seems so, as I've just got another panic. Just curious, what uses that
much memory in those moments?
Here are last contents of ssh consoles:
---
kstat.zfs.misc.arcstats.size: 804244704
kstat.zfs.misc.arcstats.size: 804244704
kstat.zfs.misc.arcstats.size: 804244704
---
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr ad8 ad10 in sy cs us sy id
0 0 0 1017M 5348M 308 0 0 0 253 0 0 0 12 411 655 0 0 100
0 0 0 1017M 5348M 241 0 0 0 257 0 0 0 18 381 748 0 0 100
0 0 0 1017M 5348M 308 0 0 0 253 0 0 0 16 411 645 0 0 100
---
last pid: 7620; load averages: 1.91, 1.80, 1.42 up 0+00:48:56 05:44:11
83 processes: 1 running, 82 sleeping
CPU: 0.2% user, 0.0% nice, 0.0% system, 0.0% interrupt, 99.8% idle
Mem: 149M Active, 60M Inact, 2317M Wired, 1292K Cache, 821M Buf, 5347M Free
Swap: 10G Total, 10G Free
---
Will give it another go with 512MB arc and 4GB kmem.
--
Dmitry Marakasov . 55B5 0596 FF1E 8D84 5F56 9510 D35A 80DD F9D2 F77D
amdmi3 at amdmi3.ru ..: jabber: amdmi3 at jabber.ru http://www.amdmi3.ru
More information about the freebsd-fs
mailing list