Memory accounting discrepancy in top (possibly ZFS related)
Peter Schuller
peter.schuller at infidyne.com
Mon Sep 10 13:40:47 PDT 2007
Hello,
(I realize I am posting alot to -current lately. If people feel I am
too quick to flood the lists let me know, but I do try to keep to
things that seem relevant for -current.)
I have a machine (Dell 2950) where I orignally thought there was a
memory visibility problem. It has 4 GB in total which seemed to be
detected on boot based on dmesg, yet top showed a total of slightly
above 2 GB. I posted to -questions about this in the belief that this
was the case from boot onwards.
However, I discovered that after boot memory adds up in top, but then
goes down. Observe the following progression of top memory and their
totals over the course of about a day, on amd64 with CURRENT from
2007-09-09:
Mem: 10M Active, 1388K Inact, 84M Wired, 268K Cache, 800K Buf, 3822M Free
Total: 3918
Mem: 272M Active, 31M Inact, 650M Wired, 42M Cache, 992K Buf, 1391M Free
Total: 2386.9
Mem: 274M Active, 32M Inact, 655M Wired, 52M Cache, 992K Buf, 1372M Free
Total: 2385.9
Mem: 282M Active, 31M Inact, 646M Wired, 53M Cache, 992K Buf, 1372M Free
Total: 2384.9
Mem: 332M Active, 146M Inact, 719M Wired, 52M Cache, 214M Buf, 1071M Free
Total: 2534
Possibly related is that on this machine I also recently saw a strange
situation where the amount of "inactive" memory was very high (2.5 gb)
when I would have expected 10-400 mb or something along those
lines. Even stranger is that if I then ran a memory grabbing process,
that memory would go from inactive to active, and then back to FREE
when the program terminated. This happened WITHOUT any disk I/O
(swapping). This was the reason why I cvsup:ed the machine on -09.
My understanding is that memory that is 'inactive' could only be freed
for use by another process by either the memory being de-allocated, or
going into swap. Thus I am/was unable to explain how the memory could
be pushed from inactive to free, by running a process that temporarily
gobbled up a few hundred megs of RAM, without seeing corresponding
disk I/O.
This situation occurred after I had dd:ed 8 gigabytes of zeros onto a
file on UFS, mdconfig attached it, and swapon:ed it. It may or may not
have been like this immediately before I did this; I do not know for
sure. This was after an otherwise fresh boot with nothing big running,
which is why the 2.5 GB seemed like a huge number.
loader.conf on this machine:
vm.kmem_size_max="1258291200"
vfs.zfs.prefetch_disable="1"
vfs.zfs.arc_max="838860800"
geom_label_load="YES"
geom_mirror_load="YES"
zfs_load="YES"
vfs.root.mountfrom="zfs:tank/root"
kern.ipc.semmni=256
kern.ipc.semmns=512
kern.ipc.semmnu=256
vmstat -m on the machine as it appears now with the top discrepancy,
but NOT necessarily matching the time of the unaccounted-for
"inactive" memory. Note that 'solaris' seems higher than it should be
given arc_max:
Type InUse MemUse HighUse Requests Size(s)
DEVFS2 160 10K - 568 16,32,64
DEVFS_RULE 34 16K - 34 64,512
DEVFS 60 2K - 61 16,128
ntfs_nthash 1 512K - 1
pfs_nodes 20 5K - 20 256
GEOM 256 44K - 135264 16,32,64,128,256,512,1024,2048
ata_dma 2 1K - 2 256
isadev 7 1K - 7 128
acd_driver 1 2K - 1 2048
cdev 23 6K - 23 256
sigio 1 1K - 1 64
filedesc 415 328K - 130951 16,32,512,1024,2048,4096
kenv 76 11K - 82 16,32,64
kqueue 120 135K - 164250 256,2048
proc-args 180 9K - 166624 16,32,64,128,256
ithread 96 18K - 96 32,128,256
prison 9 9K - 9 16,64,2048
CAM dev queue 1 1K - 1 128
KTRACE 100 13K - 100 128
CAM queue 3 1K - 3 16
linker 125 260K - 164 16,32,64,128,256,512,1024,2048
lockf 44 6K - 1473726 128
acpisem 15 2K - 15 128
ip6ndp 4 1K - 4 64,128
temp 30 272K - 186563 16,32,64,128,256,512,1024,2048,4096
devbuf 16748 35139K - 16847 16,32,64,128,256,512,1024,2048,4096
module 376 47K - 376 128
mtx_pool 1 8K - 1
subproc 854 1502K - 88424 512,4096
proc 2 16K - 2
session 88 11K - 18149 128
pgrp 104 13K - 21773 128
cred 279 70K - 2578543 256
uidinfo 19 4K - 7194 64,2048
plimit 98 25K - 87992 256
kbdmux 6 9K - 6 16,256,512,2048,4096
sysctltmp 0 0K - 55695 16,32,64,128
sysctloid 4144 208K - 4194 16,32,64,128
sysctl 0 0K - 34582 16,32,64
umtx 592 74K - 592 128
p1003.1b 1 1K - 1 16
SWAP 2 1097K - 2 64
bus-sc 87 88K - 3128 16,32,64,128,256,512,1024,2048,4096
bus 1019 81K - 22677 16,32,64,128,256,1024
devstat 28 57K - 28 32,4096
eventhandler 77 7K - 77 64,128
kobj 235 940K - 5123 4096
md_disk 1 2K - 1 2048
mfibuf 6 139K - 32 32,256,512,2048
rman 114 14K - 584 16,64,128
sbuf 0 0K - 39534 16,32,64,128,256,512,1024,2048,4096
CAM SIM 1 1K - 1 256
CAM periph 2 1K - 9 16,32,64,128,256
taskqueue 9 1K - 9 16,32,128
Unitno 9 1K - 55 32,64
iov 0 0K - 833411 16,64,128,256,512,1024,2048
ioctlops 0 0K - 341736 16,32,64,128,256,512,1024,2048,4096
msg 4 30K - 4 2048,4096
sem 4 76K - 4
shm 1 16K - 1
ttys 3696 502K - 16667 128,1024
ptys 23 6K - 23 256
acpidev 54 4K - 54 64
mbuf_tag 0 0K - 4322792 32,64
pcb 50 154K - 75751 16,32,64,128,4096
soname 77 9K - 1302904 16,32,128
biobuf 0 0K - 3 2048
vfscache 1 1024K - 1
vfs_hash 1 512K - 1
vnodes 19 1K - 78 32,256
vnodemarker 0 0K - 105700 512
mount 198 8K - 357 16,32,64,128,256
CAM XPT 10 2K - 25 32,64,128,1024
BPF 3 1K - 3 128
ether_multi 15 1K - 16 16,32,64
ifaddr 55 17K - 55 32,64,128,256,512,2048,4096
ifnet 4 7K - 4 256,2048
clone 5 20K - 5 4096
arpcom 2 1K - 2 16
lo 1 1K - 1 32
routetbl 39 7K - 23058 32,64,128,256,512
in_multi 2 1K - 2 64
sctp_iter 0 0K - 6 256
sctp_ifn 2 1K - 2 128
sctp_ifa 7 1K - 7 128
sctp_vrf 1 1K - 1 64
sctp_a_it 0 0K - 6 16
hostcache 1 36K - 1
tcplog 0 0K - 3341 256
syncache 1 100K - 1
in6_multi 16 1K - 16 32,64,128
pci_link 16 2K - 16 32,128
nfss_daemon 1 16K - 1
audit_evclass 150 5K - 187 32
newblk 1 1K - 1 512
inodedep 1 512K - 1
pagedep 1 128K - 1
ufs_dirhash 771 150K - 771 16,32,64,128,256,512
ufs_mount 6 21K - 6 512,2048,4096
UMAHash 3 14K - 12 512,1024,2048,4096
vm_pgdata 2 129K - 2 128
memdesc 1 4K - 1 4096
acpica 843 74K - 26775 16,32,64,128,256,512,1024
acpitask 0 0K - 1 64
io_apic 2 4K - 2 2048
ata_generic 1 1K - 1 1024
msi 2 1K - 2 128
nexusdev 4 1K - 4 16
entropy 1024 64K - 1024 64
atkbddev 2 1K - 2 64
USBdev 21 8K - 21 16,32,128,256,512
USB 87 19K - 102 16,32,64,128,256,1024
DEVFS1 160 80K - 164 512
DEVFS3 917 230K - 938 256
zones_data 4 1K - 4 16
solaris 1163491 949421K - 217472361 16,32,64,128,256,512,1024,2048,4096
kstat_data 1 1K - 1 64
mirror_data 4 2K - 31 64,256,512
linux 14 1K - 14 16,64
ZFS related sysctls (grep zfs):
vfs.zfs.arc_min: 39321600
vfs.zfs.arc_max: 838860800
vfs.zfs.mdcomp_disable: 0
vfs.zfs.prefetch_disable: 1
vfs.zfs.zio.taskq_threads: 0
vfs.zfs.recover: 0
vfs.zfs.vdev.cache.size: 10485760
vfs.zfs.vdev.cache.max: 16384
vfs.zfs.cache_flush_disable: 0
vfs.zfs.zil_disable: 0
vfs.zfs.debug: 0
kstat.zfs.misc.arcstats.hits: 12088667
kstat.zfs.misc.arcstats.misses: 769430
kstat.zfs.misc.arcstats.demand_data_hits: 3589717
kstat.zfs.misc.arcstats.demand_data_misses: 46936
kstat.zfs.misc.arcstats.demand_metadata_hits: 8498950
kstat.zfs.misc.arcstats.demand_metadata_misses: 722494
kstat.zfs.misc.arcstats.prefetch_data_hits: 0
kstat.zfs.misc.arcstats.prefetch_data_misses: 0
kstat.zfs.misc.arcstats.prefetch_metadata_hits: 0
kstat.zfs.misc.arcstats.prefetch_metadata_misses: 0
kstat.zfs.misc.arcstats.mru_hits: 3374213
kstat.zfs.misc.arcstats.mru_ghost_hits: 6831
kstat.zfs.misc.arcstats.mfu_hits: 8714454
kstat.zfs.misc.arcstats.mfu_ghost_hits: 66142
kstat.zfs.misc.arcstats.deleted: 576876
kstat.zfs.misc.arcstats.recycle_miss: 52143
kstat.zfs.misc.arcstats.mutex_miss: 1173
kstat.zfs.misc.arcstats.evict_skip: 61
kstat.zfs.misc.arcstats.hash_elements: 145695
kstat.zfs.misc.arcstats.hash_elements_max: 150128
kstat.zfs.misc.arcstats.hash_collisions: 2187962
kstat.zfs.misc.arcstats.hash_chains: 42687
kstat.zfs.misc.arcstats.hash_chain_max: 12
kstat.zfs.misc.arcstats.p: 479967938
kstat.zfs.misc.arcstats.c: 628830540
kstat.zfs.misc.arcstats.c_min: 39321600
kstat.zfs.misc.arcstats.c_max: 838860800
kstat.zfs.misc.arcstats.size: 628438016
So once again I don't have debugging information/traces available,
because I am not sure what would be wanted. If asked for I will
provide anything I can provide given the circumstances.
--
/ Peter Schuller
PGP userID: 0xE9758B7D or 'Peter Schuller <peter.schuller at infidyne.com>'
Key retrieval: Send an E-Mail to getpgpkey at scode.org
E-Mail: peter.schuller at infidyne.com Web: http://www.scode.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-current/attachments/20070910/eee94582/attachment.pgp
More information about the freebsd-current
mailing list