FreeBSD using swap even though there's a lot of free memory

[doug]

On Thu, 15 Oct 2020, Jon Schneider wrote:

> top -w -oswap
>
> seems to report the right thing. Would be interesting if it's _not_ MySQL.
>
> Jon
>
> On 15/10/2020 15:23, Twingly Customer Support wrote:
>> Hi,
>> 
>> We have a server running FreeBSD 12.1-RELEASE-p10. We currently have a 
>> problem where FreeBSD starting to swap when running ZFS scrub, even though 
>> we have ~70G of free memory.
>> This did not happen before when running FreeBSD 11.3 for example. It 
>> started happening at approximately the time we upgraded from 
>> 12.1-RELEASE-p5 to 12.1-RELEASE-p6, but if the upgrade is the cause of the 
>> problem is unclear, though FreeBSD never swapped for us before that. 
>> "Laundry" memory was not something we saw before either, it started to 
>> appear at the same time as FreeBSD started swapping.
>> 
>> Eventually, after scrubbing a few times, the swap becomes full and we start 
>> seeing "swap_pager_getswapspace(24): failed" etc. in dmesg.
>> This is the memory usage a while after scrubbing, note the values for 
>> Mem/Free and Swap:
>> 
>> ```
>> % top | head -n 7
>> last pid:  8112;  load averages:  1.82,  1.77,  1.73  up 6+01:37:42 
>> 10:53:48
>> 35 processes:  1 running, 34 sleeping
>> CPU:  4.9% user,  0.0% nice,  4.2% system,  0.2% interrupt, 90.7% idle
>> Mem: 110G Active, 27G Inact, 5413M Laundry, 39G Wired, 68G Free
>> ARC: 34G Total, 28G MFU, 4101M MRU, 53M Anon, 1317M Header, 225M Other
>>       30G Compressed, 53G Uncompressed, 1.77:1 Ratio
>> Swap: 8192M Total, 6434M Used, 1757M Free, 78% Inuse
>> ```
>> 
>> We are running MySQL, which has been configured to use ~50% of the total 
>> amount memory (using innodb_buffer_pool_size=127748M)
>> ZFS ARC has been configured to use 25% of the total memory (using 
>> vfs.zfs.arc_max="63874M")
>> 
>> We have tried raising both vfs.zfs.arc_max and innodb_buffer_pool_size, but 
>> this did not make any change to the total memory usage, the free memory 
>> stays at around 70G and FreeBSD still started swapping.
>> It's as if the memory is capped at around 180G for some reason.
>> 
>> Are there any configuration values that could cause FreeBSD to swap even 
>> though there's free memory? Are there any config values one could try to 
>> change in order to get FreeBSD to use the remaining ~70G of free memory 
>> instead of swapping?
>> 
>> Let me know if there's any more details you want me to provide and I'll 
>> attach those.
>> 
>> Thanks!
>> 
>> // Mattias

I see similar things. The Jails in question are 11.1. The systems updated 
to 12.1 do not display this behavior. This 11.1 system runs 5 jails. 
Swapinfo is shown below:

Device          1K-blocks     Used    Avail Capacity
/dev/aacd0p3      4194304  1776000  2418304    42%

These numbers are developed from top on the base system

[ 0 50861 ]  root
[ 20 281903 ]  camden         squirellmail/roundcube/postfix/mysql
[ 21 322759 ]  bassharbor     wordpress/php56
[ 19 343522 ]  monhegan       wordpress/php56
[ 18 369139 ]  newharbor      apache24/sendmail
[ 17 587332 ]  pemaquid       wordpress/php74

Jails:  1904655
total:  1955516

I read somewhere that the virtual memory system pre-pages modified pages as 
a just-in-case measure. If this is correct, 12.1 does not do this on a 
non-paging system. The system shown above uses about 10% swapspace after a 
reboot and works its way to the 42% shown above in a day or so.