lightly loaded system eats swap space

Tue Jun 19 19:29:46 UTC 2018

Hi
I had something similar on FreeNAS 11.1 ( based on FreeBSD 11.1).
Swap was allocated and never released until it ran out.
Some time ago I set the following sysctl:
vm.disable_swapspace_pageouts: 1

That completely stopped swap allocation and I have not rebooted since, except for OS patching.

From what I found using google this sysctl may have some nasty side effects when system runs out of memory,
but that has not happened on my system.

	Paul

> On 19 Jun 2018, at 19:57, Cassiano Peixoto <peixotocassiano at gmail.com> wrote:
> 
> Hi,
> 
> I have the very same issue on many servers. Mainly on mail servers running
> qmail+spamassassin+clamav. I've tuned some variables on loader.conf:
> 
> vfs.zfs.vdev.cache.size="2G"
> vfs.zfs.arc_min="614400000"
> vfs.zfs.arc_max="4915200000"
> 
> But after some days, it begins eating swap and the system comes very very
> slow then I need to reboot it.
> 
> My system config is:
> 
> FreeBSD 11.1-STABLE #5 r321625M: Thu Sep 21 16:01:56 -03 2017
>    root at mail.com:/usr/obj/usr/src/sys/MAIL amd64
> FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM
> 4.0.0)
> VT(vga): resolution 640x480
> CPU: Intel(R) Atom(TM) CPU  C2518  @ 1.74GHz (1750.04-MHz K8-class CPU)
>  Origin="GenuineIntel"  Id=0x406d8  Family=0x6  Model=0x4d  Stepping=8
> 
> Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
> 
> Features2=0x43d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,AESNI,RDRAND>
>  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
>  AMD Features2=0x101<LAHF,Prefetch>
>  Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG>
>  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
>  TSC: P-state invariant, performance statistics
> real memory  = 8589934592 (8192 MB)
> avail memory = 8213245952 (7832 MB)
> 
> It's configured with 4GB of swap. Let me know if I can help with any other
> information.
> 
> Thanks.
> 
> 
> On Tue, Jun 19, 2018 at 2:29 PM, Jeremy Chadwick <jdc at koitsu.org> wrote:
> 
>> (I am not subscribed to -stable, so please CC me, though I doubt I can
>> help in any way/shape/form past this Email)
>> 
>> Not the first time this has come up -- and every time it has, all that's
>> heard is crickets in the threads.  Recent proof:
>> 
>> https://lists.freebsd.org/pipermail/freebsd-stable/2018-April/088727.html
>> https://lists.freebsd.org/pipermail/freebsd-stable/2018-April/088728.html
>> https://lists.freebsd.org/pipermail/freebsd-stable/2018-June/089094.html
>> 
>> I sent private mail to Peter Jeremy about his issue.  I will not
>> disclose that Email here.  However, I will disclose the commits I
>> included in said Email that have touched ZFS ARC-related code:
>> 
>> http://www.freshbsd.org/commit/freebsd/r332785
>> http://www.freshbsd.org/commit/freebsd/r332552
>> http://www.freshbsd.org/commit/freebsd/r332540 (may help give insights)
>> http://www.freshbsd.org/commit/freebsd/r330061
>> http://www.freshbsd.org/commit/freebsd/r328235
>> http://www.freshbsd.org/commit/freebsd/r327491
>> http://www.freshbsd.org/commit/freebsd/r326619
>> http://www.freshbsd.org/commit/freebsd/r326427 (quota-related, maybe
>> irrelevant)
>> http://www.freshbsd.org/commit/freebsd/r323667
>> 
>> In short (and nebulous as hell; sorry, I cannot be more specific given
>> the nature of the problem): there have been changes about ZFS's memory
>> allocation/releasing decision-making scheme compared to ZFS on "older"
>> FreeBSD (i.e. earlier 11.x, and definitely 10.x and 9.x).
>> 
>> Recommendations like "limit your ARC" are nothing new in FreeBSD, but
>> are still ridiculous kludges: tech-lists' system clearly has 105GB MRU
>> (MRU = most recently used) in ARC, meaning there is memory that can be
>> released back to the rest of the OS for general use (re: memory
>> contention/pressure situation), but the OS is choosing to use swap
>> instead, eventually exhausting it.  That logic sounds broken, IMO.  (And
>> yes I did notice the size of bhyve process)
>> 
>> ZFS-related kernel folks need to be involved in this conversation.  For
>> whatever reason, in the past several years, related committers are no
>> longer participating in these type of discussions.  The opposite was
>> true back in the 7.x to 9.x days.  The answers have to come from them.
>> I don't know, today, a) how they prefer these problems get reported to
>> them, or b) what exact information they want that can help narrow it
>> down (tech-lists' provided data is, IMO, good and par for the course).
>> 
>> --
>> | Jeremy Chadwick                                   jdc at koitsu.org |
>> | UNIX Systems Administrator                http://jdc.koitsu.org/ |
>> | Making life hard for others since 1977.             PGP 4BD6C0CB |
>> 
>> _______________________________________________
>> freebsd-stable at freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
>> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
>> 
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
> 