Tracking down what has inact pages locked up

[Karl Denninger]

Given the following:

     3 users    Load  2.66  2.46  2.37                  Mar 16 15:26

Mem:KB    REAL            VIRTUAL                       VN PAGER   SWAP PAGER
         Tot   Share      Tot    Share    Free           in   out     in   out
Act  879332   19864  8732852    54112  472560  count                        5
All 1681232   26376  9557840   218308          pages                       13
Proc:                                                            Interrupts
   r   p   d   s   w   Csw  Trp  Sys  Int  Sof  Flt        ioflt 25470 total
   3         206  48   71k  28k 153k  15k  279 7015   1202 cow      11 uart0 4
                                                      1487 zfod   1647 uhci0 16
  6.7%Sys   1.9%Intr  4.0%User  0.0%Nice 87.3%Idle     122 ozfod       pcm0 17
|    |    |    |    |    |    |    |    |    |          8%ozfod       ehci0 uhci
===+>>                                                    daefr       uhci1 21
                                         73 dtbuf     2063 prcfr   523 uhci3 ehci
Namei     Name-cache   Dir-cache    485946 desvn     6426 totfr   800 arcmsr0 30
    Calls    hits   %    hits   %    163373 numvn          react  1065 cpu0:timer
     8636    8554  99       2   0    121487 frevn          pdwak   220 mps0 256
                                                    833329 pdpgs  6090 em0:rx 0
Disks   da0   da1   da2   da3   da4   da5   da6           intrn  5759 em0:tx 0
KB/t  11.07 63.91 14.88 20.64 16.49 14.03  0.00   4801680 wire        em0:link
tps       6  1222    25    81    73    26     0    608344 act      67 em1:rx 0
MB/s   0.06 76.26  0.36  1.63  1.17  0.35  0.00  18579164 inact    69 em1:tx 0
%busy     0    20     7    38    35     8     0    472216 cache       em1:link
                                                      3224 free        ahci0:ch0
                                                   1694896 buf         ahci0:ch2
                                                                   541 cpu1:timer
                                                                   744 cpu11:time
                                                                   515 cpu2:timer
                                                                   673 cpu10:time
                                                                   811 cpu5:timer
                                                                   955 cpu13:time
                                                                   473 cpu4:timer
                                                                   442 cpu12:time
                                                                   542 cpu3:timer
                                                                   524 cpu8:timer
                                                                   530 cpu6:timer
                                                                   546 cpu9:timer
                                                                   540 cpu7:timer
                                                                   940 cpu14:time
                                                                   443 cpu15:time

Note the "free" and "inact" counts.

The system ought, if I'm reading the sysctl variables correctly, be 
attempting to aggressively reclaim that memory.

vm.v_free_min: 38595
vm.v_free_target: 130338
vm.v_free_reserved: 8014
vm.v_inactive_target: 195507
vm.v_cache_min: 0
vm.v_cache_max: 0
vm.v_pageout_free_min: 34
vm.swap_enabled: 1

It's not.

In fact, what's happening is that it is (slowly) filling the swap!

There is nothing in the RSS or size of the processes that give 
indication of a rogue process that has run away with all the memory, 
never mind that it obviously isn't being touched or it would be in the 
"active" bucket.  Shutting down the user processes (this is a pretty 
busy box) does not clear any of it out.

Is there a reasonable way to determine who or what has that memory 
locked up -- and thus why the vm system is not demoting that space into 
the cache bucket so it can be freed (which, if my understanding is 
correct, should be happening long before now!)

The system has both ZFS and UFS filesystems on it.  I did put the PR on 
there I submitted to change the ARC cache sizing to dynamic, and it is 
(correctly) sizing down the ARC cache over time to the point that it is 
now pinned on the minimum due to low RAM.

This is what I have loaded as modules:

Id Refs Address            Size     Name
  1   36 0xffffffff80200000 166de08  kernel
  2    1 0xffffffff8186e000 20400    geom_eli.ko
  3    1 0xffffffff8188f000 6dc8     aesni.ko
  4    1 0xffffffff81a12000 15af3a   zfs.ko
  5    1 0xffffffff81b6d000 3847     opensolaris.ko
  6    1 0xffffffff81b71000 34d3     ums.ko
  7    1 0xffffffff81b75000 7c18     uftdi.ko
  8    1 0xffffffff81b7d000 5134     ucom.ko
  9    1 0xffffffff81b83000 2a44     uhid.ko
10    2 0xffffffff81b86000 10a42    ipfw.ko
11    1 0xffffffff81b97000 478d     ipfw_nat.ko
12    1 0xffffffff81b9c000 daf2     libalias.ko
13    1 0xffffffff81baa000 1f4b     daemon_saver.ko

FreeBSD 10.0-STABLE #13 r263037M: Fri Mar 14 14:58:11 CDT 2014 
karl at NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP

-- 
-- Karl
karl at denninger.net


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2711 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20140316/242a58d5/attachment.bin>