Fwd: Wired Memory Increasing about 500MBytes per day

From: Özkan KIRIK <ozkan.kirik_at_gmail.com>
Date: Tue, 03 Aug 2021 04:01:47 UTC
I'm sorry for cross posting, but I couldn't received any replies.

---------- Forwarded message ---------
From: Özkan KIRIK <ozkan.kirik@gmail.com>
Date: Mon, Aug 2, 2021 at 8:00 AM
Subject: Wired Memory Increasing about 500MBytes per day
To: FreeBSD Net <freebsd-net@freebsd.org>, freebsd-stable <
freebsd-stable@freebsd.org>


Hello,

I'm using FreeBSD stable/12 0f97f2a1857a96563792f0d873b11a16ff9f818c (Jul
25) built.
pf, ipfw and ipsec options are built with kernel. The server is used as
firewall that squid and snort3 (daq - netmap) is running.

I saw that, wired memory is increasing every day. It's about 500MBytes per
day. I'm checking vmstat and top (sorted by res), I couldn't find what is
consuming the wired memory.

How can I find that which process or which part of kernel is consuming the
wired memory ?

# first a few lines of top
last pid: 61147;  load averages:  3.83,  3.92,  4.02    up 3+14:09:57
 07:53:44
4294 threads:  42 running, 4153 sleeping, 99 waiting
CPU:  3.1% user,  0.0% nice, 10.4% system,  0.0% interrupt, 86.5% idle
Mem: 3074M Active, 11G Inact, 11G Wired, 1572M Buf, 101G Free
ARC: 3764M Total, 1385M MFU, 2144M MRU, 2666K Anon, 19M Header, 213M Other
     3243M Compressed, 3510M Uncompressed, 1.08:1 Ratio

# vmstat -m
         Type InUse MemUse HighUse Requests  Size(s)
CAM dev queue     5     1K       -        5  64
      scsi_da     0     0K       -      388  32,256
         UART     3     3K       -        3  16,1024
          USB    43    87K       -       78
 16,32,128,256,512,1024,2048,4096,16384,32768
       USBdev    31     4K       -       53  32,64,128,256,512,4096
     SCSI ENC    26   133K       -    67266  16,64,512,1024,2048,32768,65536
       nvlist   282    26K       - 52522369  16,32,64,128,256,2048,4096,8192
        vtbuf    24  1968K       -       46  4096
           vt    11     6K       -       11  512
     acpiintr     1     1K       -        1  64
       DEVFS3   194    49K       -      269  256
       DEVFS1   146    73K       -      217  512
   DEVFS_RULE    56    27K       -       56  64,512
        DEVFS    15     1K       -       20  16,32,128
       DEVFSP   529    34K       - 748886191  64
  nullfs_hash     1 16384K       -        1
  nullfs_node    27     2K       -       34  64
 nullfs_mount    10     1K       -       11  32
    pfs_nodes    20    10K       -       20  512
  tmpfs mount     3     1K       -        3  128
   tmpfs name  4791   116K       -    18569  16,32,64,128
         GEOM   357    54K       -     3904
 16,32,64,128,256,512,1024,2048,4096,8192,16384
    raid_data     0     0K       -      612  32,128,256
       isadev     7     1K       -        7  128
       acpica 16568  1655K       -   738465
 16,32,64,128,256,512,1024,2048,4096
     acpitask     1    64K       -        1  65536
      acpisem    24     3K       -       24  128
         cdev     4     1K       -        4  256
     filedesc   367  2390K       -  4457314
 16,32,64,128,256,512,4096,8192,16384,32768,65536
        sigio    17     2K       -      415  64
     filecaps     6     1K       -    90314  16,32,64
      kdtrace  4829  1110K       - 15580812  64,256
         kenv   148    14K       -      148  16,32,64,128,8192
       kqueue  1176   726K       - 25696080
 64,128,256,512,2048,4096,8192,16384,32768
    proc-args   634    46K       -  6581654  16,32,64,128,256
        hhook    34     7K       -       51  32,256
      ithread   449    73K       -      455  32,128,256
       prison    11     5K       -       12  16,32,4096
       KTRACE   100    13K       -      100  128
        evdev     3     3K       -        7  1024
       linker   481   505K       -      633
 16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
        lockf    39     5K       -  7173761  64,128
   loginclass     3     1K       -        8  64
       devbuf 19975 49482K       -    38585
 16,32,64,128,256,512,1024,2048,4096,8192,16384,65536
         temp  4192   173K       - 75602914
 16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
       module   537    68K       -      538  128
     mtx_pool     2    72K       -        2  8192,65536
          osd    14     1K       -   129800  16,32,64,128,256
     pmchooks     1     1K       -        1  128
          pmc     2     1K       -        2  64
      session    93    12K       -   437777  128
         proc     2   256K       -        2
      subproc  1468  2561K       -  6842581  512,4096
         cred  1869   468K       -  2482324  256
       plimit    83    21K       -   827841  256
      uidinfo    14    34K       -    17233  128,32768
           ix    42   117K       -       42  512,1024,2048,4096
       sysctl     1     1K       -  2408297  32,64
    sysctloid 15990   820K       -    16624  16,32,64,128,256
    sysctltmp     0     0K       -  5790925  16,64,256,1024
      acpidev   179    12K       -      179  64
      CAM SIM     6     2K       -        6  256
      tidhash     1   256K       -        1
      callout    33  9352K       -       33
         umtx 10204  1276K       -    10204  128
     p1003.1b     1     1K       -        1  16
          bus  2040   231K       -   341181  16,32,64,128,256,512,1024,4096
       bus-sc   202  1751K       -    72634
 16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
      devstat    20    41K       -       20  32,4096
 eventhandler   169    14K       -      170  64,128
     firmware    26     2K       -       26  16,32,128
   gtaskqueue   198    44K       -      198  16,32,256,8192
         kobj   332  1328K       -     1289  4096
      Per-cpu     1     1K       -        1  32
      CAM XPT    42     4K       -     1443
 16,32,64,128,256,512,1024,2048,65536
         rman   408    48K       -      816  16,32,128
         sbuf     1     1K       -   839349
 16,32,64,128,256,512,2048,4096,8192,16384,32768,65536
    toponodes    84    11K       -       84  128
      CAM DEV     9    18K       -      520  2048
    taskqueue   648    74K       -      942  16,32,64,128,256,512
     terminal    11     3K       -       11  256
       Unitno    65     4K       -  3793393  32,64
         vmem     4  2368K       -       12  4096,8192,16384,32768,65536
     ioctlops     0     0K       -  2234885  256,512,1024,2048,4096
       select  2671   334K       -     2671  128
          iov     0     0K       - 248581529  16,64,128,256,512,1024,4096
          msg     4   204K       -        4  4096,8192,65536
          sem     4   106K       -        4  2048,4096
          shm    14    58K       -   282515  2048,32768
          tty    14    14K       -       24  1024
          pts     1     1K       -       11  256
     mbuf_tag     0     0K       - 339649313  32,64
        shmfd    34    23K       -    21424  64,256,1024,8192
       soname    78     7K       - 49413750  16,32,64,128
          pcb   990  4460K       -   669045  16,32,64,128,1024,2048,8192
       kbdmux     6    22K       -        8  16,512,1024,2048,16384
          acl     0     0K       -   124531  4096
     vfscache     4 32961K       -        4  256,65536
   cl_savebuf     0     0K       -      247  64
     vfs_hash     1 16384K       -        1
       vnodes     1     1K       -        1  256
        mount   261    13K       -    61165  16,32,64,128,256,1024
       statfs     0     0K       -  3613124  4096
          LED     4     1K       -        4  16,128
  vnodemarker     0     0K       -   214729  512
      fadvise     0     0K       -      137  32
chacha20random     1     8K       -        1  8192
          BPF  2535 25717K       - 2247023944  16,64,128,256,512,2048,4096
      ifdescr     1     1K       -        1  16
        ifnet   513  1027K       -     1482  64,128,256,512,1024,2048,4096
       ifaddr  5117  1388K       -    15096  16,32,64,128,256,512,2048,4096
  ether_multi 13633  1089K       -    40471  16,32,64,128
        clone    38     5K       -       55  128
        epair     2     1K       -        4  128
        ipsec     6     2K       -        9  256
      lltable 37732 18363K       -   246041  256,512
          tun     7     1K       -       11  32,256
         vlan  2864   359K       -    38902  64,128,256
        iflib   314  3652K       -      338
 32,64,256,512,1024,4096,8192,16384,32768
     routetbl 12731  3113K       -   247870  32,64,128,256,512,8192
         vnet     2     1K       -        3  64
    vnet_data     2   384K       -        3
vnet_data_free     1     1K       -        1  32
         igmp   511    64K       -     1479  128
     in_multi   487   122K       -     1446  256
  ip_moptions     0     0K       -        6  64
encap_export_host    18     1K       -       18  32,64
    mroutetbl     8    53K       -       12  256,2048,8192,16384
    sctp_a_it     0     0K       -     6133  16
     sctp_vrf     2     1K       -        3  64
     sctp_ifa  1572   197K       -     6888  128
     sctp_ifn   487    61K       -     6888  128
    sctp_iter     0     0K       -     6133  256
   tfo_ccache     2   256K       -        3
    hostcache     2    64K       -        3  32768
          LRO    48   960K       -       48  8192,32768
      tcpfunc     1     1K       -        1  64
     syncache     2  1088K       -        3
    in6_multi  4880   686K       -    14475  32,256
 ip6_moptions     0     0K       -        1  32
          mld   497    63K       -     1461  128
       ip6ndp   983   153K       -     4354  64,256
  inpcbpolicy  6117   192K       - 952852053  32
     secasvar     8     4K       -     1390  256,1024
       sahead     8     4K       -     1052  256,1024
  ipsecpolicy    56    16K       -     1530  256,1024
 ipsecrequest    44     6K       -     1516  128
   ipsec-misc    51     2K       -     8317  16,32
    ipsec-saq     4     4K       -      152  256,1024
    ipsec-reg     2     1K       -        2  32
     dummynet    10  2069K       -       13  512,1024
     dummynet     8     3K       -       10  256,512
  IpFw/IpAcct    94  2428K       -     1721
 16,32,64,128,256,512,1024,2048,4096,8192
     ipfw_tbl     3     1K       -      267  128
       pfsync     4    65K       -        6  512,32768
      pf_temp     0     0K       -    10651  32,64
      pf_hash    14 23056K       -       21  2048
     pf_ifnet   543   140K       -    30387  128,256,2048
      pf_rule  6475  9594K       -    70452  128,2048
      pf_osfp  2382   245K       -     7146  64,128
     pf_table  1647  3294K       -    56328  2048
        linux     5     1K       -        6  16,64
 linuxcurrent     2     1K       -        2  64,256
       crypto    58    43K       -     3734  128,512,1024,4096
        xform     0     0K       - 145609856  64
      pagedep     1  2048K       -        4  256
     inodedep     1 16384K       -        6  512
    bmsafemap     1     8K       -        5  256,8192
       newblk     1 32768K       -        4  256
     freeblks     0     0K       -        1  128
     freefile     0     0K       -        1  64
       diradd     0     0K       -        8  128
        mkdir     0     0K       -        4  128
       dirrem     0     0K       -        5  128
    newdirblk     0     0K       -        2  64
     freework     1     1K       -        2  16,128
        sbdep     0     0K       -        3  64
     savedino     0     0K       -        4  256
      softdep     1     1K       -        1  512
  ufs_dirhash   600   113K       -     1164  16,32,64,128,256,512
    ufs_mount     9    50K       -       68  512,4096,8192
      UMAHash    61  9148K       -      155
 512,1024,2048,4096,8192,16384,32768,65536
      md_disk    17     5K       -       17  32,4096
   md_sectors    16    64K       -       16  4096
          rpc     1     4K       -        1  4096
  fpukern_ctx    32    32K       -       32  1024
      memdesc     1     4K       -        1  4096
     pci_link    16     2K       -       16  64,128
      CAM CCB     0     0K       - 47838681  2048
     mrsasbuf   949  2860K       -      951  32,128,256,2048,8192
     CAM path    13     1K       -     7263  32
   CAM periph    10     3K       -      541  16,32,64,128,256
       netmap 262944 1126176K       -   262944
    acpi_perf    32    16K       -       32  512
       sr_iov     6     3K       -        6  512
       apmdev     1     1K       -        1  128
   madt_table     0     0K       -        2  256,4096
CAM I/O Scheduler     2     1K       -        2  128
      entropy     1     1K       -  5527116  32,4096
         intr     4   484K       -        4  65536
      io_apic     3     6K       -        3  2048
   local_apic     1    32K       -        1  32768
    CAM queue    14    52K       -     1551
 16,32,64,128,256,512,1024,2048,4096,16384
          MCA    65    21K       -       65  128,256,512
         cpus     2     1K       -        2  128
          msi    87    11K       -       87  128
     nexusdev     6     1K       -        6  16
      scsi_cd     0     0K       -        8  16
   kstat_data    12     1K       -       12  64
      solaris 366461 463423K       - 2105003083
 16,32,64,128,256,512,1024,2048,4096,8192,16384,32768
    sfs_nodes     6     3K       -        6  512
   aesni_data     2     3K       -        2  256,2048
         ioat    16  7168K       -       16
         disc     1     1K       -        2  16
          gre     5     1K       -        5  64,128
 netgraph_msg     0     0K       -        5  64,128,256
netgraph_hook     0     0K       -        2  128
netgraph_node     7     2K       -       11  128,256
netgraph_path     0     0K       -        4  16
netgraph_sock     6     1K       -        9  32,128
netgraph_mppc     0     0K       -        2  1024,65536
     eli data    35     4K       -   865337
 64,256,1024,2048,4096,8192,16384,32768,65536

Regards,