lightly loaded system eats swap space

[Cassiano Peixoto]

Hi,

I have the very same issue on many servers. Mainly on mail servers running
qmail+spamassassin+clamav. I've tuned some variables on loader.conf:

vfs.zfs.vdev.cache.size="2G"
vfs.zfs.arc_min="614400000"
vfs.zfs.arc_max="4915200000"

But after some days, it begins eating swap and the system comes very very
slow then I need to reboot it.

My system config is:

FreeBSD 11.1-STABLE #5 r321625M: Thu Sep 21 16:01:56 -03 2017
    root at mail.com:/usr/obj/usr/src/sys/MAIL amd64
FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM
4.0.0)
VT(vga): resolution 640x480
CPU: Intel(R) Atom(TM) CPU  C2518  @ 1.74GHz (1750.04-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x406d8  Family=0x6  Model=0x4d  Stepping=8

Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>

Features2=0x43d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,AESNI,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 8589934592 (8192 MB)
avail memory = 8213245952 (7832 MB)

It's configured with 4GB of swap. Let me know if I can help with any other
information.

Thanks.


On Tue, Jun 19, 2018 at 2:29 PM, Jeremy Chadwick <jdc at koitsu.org> wrote:

> (I am not subscribed to -stable, so please CC me, though I doubt I can
> help in any way/shape/form past this Email)
>
> Not the first time this has come up -- and every time it has, all that's
> heard is crickets in the threads.  Recent proof:
>
> https://lists.freebsd.org/pipermail/freebsd-stable/2018-April/088727.html
> https://lists.freebsd.org/pipermail/freebsd-stable/2018-April/088728.html
> https://lists.freebsd.org/pipermail/freebsd-stable/2018-June/089094.html
>
> I sent private mail to Peter Jeremy about his issue.  I will not
> disclose that Email here.  However, I will disclose the commits I
> included in said Email that have touched ZFS ARC-related code:
>
> http://www.freshbsd.org/commit/freebsd/r332785
> http://www.freshbsd.org/commit/freebsd/r332552
> http://www.freshbsd.org/commit/freebsd/r332540 (may help give insights)
> http://www.freshbsd.org/commit/freebsd/r330061
> http://www.freshbsd.org/commit/freebsd/r328235
> http://www.freshbsd.org/commit/freebsd/r327491
> http://www.freshbsd.org/commit/freebsd/r326619
> http://www.freshbsd.org/commit/freebsd/r326427 (quota-related, maybe
> irrelevant)
> http://www.freshbsd.org/commit/freebsd/r323667
>
> In short (and nebulous as hell; sorry, I cannot be more specific given
> the nature of the problem): there have been changes about ZFS's memory
> allocation/releasing decision-making scheme compared to ZFS on "older"
> FreeBSD (i.e. earlier 11.x, and definitely 10.x and 9.x).
>
> Recommendations like "limit your ARC" are nothing new in FreeBSD, but
> are still ridiculous kludges: tech-lists' system clearly has 105GB MRU
> (MRU = most recently used) in ARC, meaning there is memory that can be
> released back to the rest of the OS for general use (re: memory
> contention/pressure situation), but the OS is choosing to use swap
> instead, eventually exhausting it.  That logic sounds broken, IMO.  (And
> yes I did notice the size of bhyve process)
>
> ZFS-related kernel folks need to be involved in this conversation.  For
> whatever reason, in the past several years, related committers are no
> longer participating in these type of discussions.  The opposite was
> true back in the 7.x to 9.x days.  The answers have to come from them.
> I don't know, today, a) how they prefer these problems get reported to
> them, or b) what exact information they want that can help narrow it
> down (tech-lists' provided data is, IMO, good and par for the course).
>
> --
> | Jeremy Chadwick                                   jdc at koitsu.org |
> | UNIX Systems Administrator                http://jdc.koitsu.org/ |
> | Making life hard for others since 1977.             PGP 4BD6C0CB |
>
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
>