Freebsd 8.0 kmem map too small

[Jeremy Chadwick]

On Tue, May 11, 2010 at 04:43:59PM +0300, Alexander V. Ribchansky wrote:
> As Artem advised, I comment out
> 
> CFLAGS+=-DZIO_USE_UMA in sys/modules/zfs/Makefile
> 
> and things like to be "back in USSR :)" if seriously - all become as good as before 18.04.2010 mega ZFS-MFC.
> While with UMA, Wired memory constantly grow up to kmem_max limit and than PANIC! :(, without it, Wired is approx 380 - 450M on typical 8-STABLE KDE3 desktop with plain ZFS.
> So or there is something wrong with my (and many many other's people's) hands or UMA broke ZFS at all.
> 
> Thank you, Artem for hint! I already start to think, that revert to pre-18.04.2010 8-STABLE is the only solution.

The interesting thing is that I'm not seeing this behaviour on either of
our boxes running RELENG_8 kernel/world dated after 2010/04/18.

Below are some stats.  I don't know if these are any help.

$ for i in localhost ra anubis; do echo ; ssh $i "uname -a ; sysctl kstat.zfs.misc.arcstats | egrep '(p|c|c_min|c_max|\.size|evict_skip|memory_throttle_count):' ; sysctl hw | egrep '(physmem|realmem|usermem):' ; egrep '(kmem_size|arc_max)=' /boot/loader.conf" ; done

FreeBSD icarus.home.lan 8.0-STABLE FreeBSD 8.0-STABLE #0: Thu Apr 22 06:11:56 PDT 2010     root at icarus.home.lan:/usr/obj/usr/src/sys/X7SBA_RELENG_8_amd64  amd64
kstat.zfs.misc.arcstats.evict_skip: 0
kstat.zfs.misc.arcstats.p: 780308480
kstat.zfs.misc.arcstats.c: 1560281088
kstat.zfs.misc.arcstats.c_min: 201326592
kstat.zfs.misc.arcstats.c_max: 1610612736
kstat.zfs.misc.arcstats.size: 1560271336
kstat.zfs.misc.arcstats.memory_throttle_count: 0
hw.physmem: 4285079552
hw.usermem: 2546024448
hw.realmem: 5100273664
vm.kmem_size="2048M"
vfs.zfs.arc_max="1536M"

FreeBSD ra 8.0-STABLE FreeBSD 8.0-STABLE #0: Mon Apr 26 02:26:36 PDT 2010     root at ra:/usr/obj/usr/src/sys/X7SBI_RELENG_8_amd64  amd64
kstat.zfs.misc.arcstats.evict_skip: 211067
kstat.zfs.misc.arcstats.p: 2633281083
kstat.zfs.misc.arcstats.c: 2968678030
kstat.zfs.misc.arcstats.c_min: 469762048
kstat.zfs.misc.arcstats.c_max: 3758096384
kstat.zfs.misc.arcstats.size: 2622834360
kstat.zfs.misc.arcstats.memory_throttle_count: 0
hw.physmem: 8580050944
hw.usermem: 2859462656
hw.realmem: 9395240960
vm.kmem_size="4096M"
vfs.zfs.arc_max="3584M"

FreeBSD anubis 8.0-STABLE FreeBSD 8.0-STABLE #0: Fri Mar 26 10:41:14 PDT 2010     root at anubis:/usr/obj/usr/src/sys/PDSMI_PLUS_RELENG_8_amd64  amd64
kstat.zfs.misc.arcstats.evict_skip: 1395
kstat.zfs.misc.arcstats.p: 1879163904
kstat.zfs.misc.arcstats.c: 3758096384
kstat.zfs.misc.arcstats.c_min: 469762048
kstat.zfs.misc.arcstats.c_max: 3758096384
kstat.zfs.misc.arcstats.size: 1421240344
kstat.zfs.misc.arcstats.memory_throttle_count: 0
hw.physmem: 8580870144
hw.usermem: 6518259712
hw.realmem: 9126805504
vm.kmem_size="4096M"
vfs.zfs.arc_max="3584M"

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |