Two odd problems with STABLE-10 r262921 [ZFS ARC memory allocation / "freeze" problems -- RESOLVED]

[Karl Denninger]

On 3/11/2014 8:50 AM, Karl Denninger wrote:
>
> The other issue is that I am running into bizarre wired memory behavior.  I
> have ZFS on the box and it was wiring down ~20GB, driving the system into
> paging.  I dropped arc_max to 16GB as a hard limit, it's currently showing
> 15GB, but...
>
> 20531152 wire
>
> zfs-stats -A shows:
>
>
> ------------------------------------------------------------------------
> ZFS Subsystem Report                            Tue Mar 11 08:27:55 2014
> ------------------------------------------------------------------------
>
> ARC Summary: (HEALTHY)
>          Memory Throttle Count:                  0
>
> ARC Misc:
>          Deleted:                                4.22m
>          Recycle Misses:                         57.10k
>          Mutex Misses:                           320
>          Evict Skips:                            2.35m
>
> ARC Size:                               95.96%  15.35   GiB
>          Target Size: (Adaptive)         100.00% 16.00   GiB
>          Min Size (Hard Limit):          12.50%  2.00    GiB
>          Max Size (High Water):          8:1     16.00   GiB
>
> ARC Size Breakdown:
>          Recently Used Cache Size:       75.49%  12.08   GiB
>          Frequently Used Cache Size:     24.51%  3.92    GiB
>
> ARC Hash Breakdown:
>          Elements Max:                           1.59m
>          Elements Current:               55.53%  883.80k
>          Collisions:                             5.84m
>          Chain Max:                              16
>          Chains:                                 261.48k
>
> ------------------------------------------------------------------------
>
> Ok, where's the rest coming from?  There's nothing in the shared memory
> segment list that appears to be related.
>
> Is there a memory leak in here somewhere?
As posted to the fs mailing list I have found and corrected this issue; 
the ARC eviction code has issues.

Please pardon the apparently-reversed patch I submitted; it's Friday, I 
didn't have enough coffee and Murphy got me twice while putting that 
together (the first time when I pulled the diff off without committing 
the last change I had made back to my local tree first!)

If you're having issues with your system stalling while running ZFS 
filesystems and it appears you're being driven into paging when you 
shouldn't be (in other words, the ARC cache is consuming all of your RAM 
and not releasing it as other things demand memory) this should fix it.  
It also includes a run-time tunable via sysctl (defaults to 25%, which 
was the original intent of the code that I replaced) for the percentage 
of memory to target as free, below which the ARC will be asked to 
release RAM.

See http://www.freebsd.org/cgi/query-pr.cgi?pr=187594

-- 
-- Karl
karl at denninger.net


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2711 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20140314/e36e4971/attachment.bin>