FreeBSD using swap even though there's a lot of free memory

[Jon Schneider]

SunOS 4 used to (speculatively) page stuff out. Probably just indicates relatively unused pages.

Jon

> On 16 Oct 2020, at 18:50, doug <doug at fledge.watson.org> wrote:
> 
> 
> 
>> On Thu, 15 Oct 2020, Jon Schneider wrote:
>> top -w -oswap
>> seems to report the right thing. Would be interesting if it's _not_ MySQL.
>> Jon
>>> On 15/10/2020 15:23, Twingly Customer Support wrote:
>>> Hi,
>>> We have a server running FreeBSD 12.1-RELEASE-p10. We currently have a problem where FreeBSD starting to swap when running ZFS scrub, even though we have ~70G of free memory.
>>> This did not happen before when running FreeBSD 11.3 for example. It started happening at approximately the time we upgraded from 12.1-RELEASE-p5 to 12.1-RELEASE-p6, but if the upgrade is the cause of the problem is unclear, though FreeBSD never swapped for us before that. "Laundry" memory was not something we saw before either, it started to appear at the same time as FreeBSD started swapping.
>>> Eventually, after scrubbing a few times, the swap becomes full and we start seeing "swap_pager_getswapspace(24): failed" etc. in dmesg.
>>> This is the memory usage a while after scrubbing, note the values for Mem/Free and Swap:
>>> ```
>>> % top | head -n 7
>>> last pid:  8112;  load averages:  1.82,  1.77,  1.73  up 6+01:37:42 10:53:48
>>> 35 processes:  1 running, 34 sleeping
>>> CPU:  4.9% user,  0.0% nice,  4.2% system,  0.2% interrupt, 90.7% idle
>>> Mem: 110G Active, 27G Inact, 5413M Laundry, 39G Wired, 68G Free
>>> ARC: 34G Total, 28G MFU, 4101M MRU, 53M Anon, 1317M Header, 225M Other
>>>     30G Compressed, 53G Uncompressed, 1.77:1 Ratio
>>> Swap: 8192M Total, 6434M Used, 1757M Free, 78% Inuse
>>> ```
>>> We are running MySQL, which has been configured to use ~50% of the total amount memory (using innodb_buffer_pool_size=127748M)
>>> ZFS ARC has been configured to use 25% of the total memory (using vfs.zfs.arc_max="63874M")
>>> We have tried raising both vfs.zfs.arc_max and innodb_buffer_pool_size, but this did not make any change to the total memory usage, the free memory stays at around 70G and FreeBSD still started swapping.
>>> It's as if the memory is capped at around 180G for some reason.
>>> Are there any configuration values that could cause FreeBSD to swap even though there's free memory? Are there any config values one could try to change in order to get FreeBSD to use the remaining ~70G of free memory instead of swapping?
>>> Let me know if there's any more details you want me to provide and I'll attach those.
>>> Thanks!
>>> // Mattias
> 
> I see similar things. The Jails in question are 11.1. The systems updated to 12.1 do not display this behavior. This 11.1 system runs 5 jails. Swapinfo is shown below:
> 
> Device          1K-blocks     Used    Avail Capacity
> /dev/aacd0p3      4194304  1776000  2418304    42%
> 
> These numbers are developed from top on the base system
> 
> [ 0 50861 ]  root
> [ 20 281903 ]  camden         squirellmail/roundcube/postfix/mysql
> [ 21 322759 ]  bassharbor     wordpress/php56
> [ 19 343522 ]  monhegan       wordpress/php56
> [ 18 369139 ]  newharbor      apache24/sendmail
> [ 17 587332 ]  pemaquid       wordpress/php74
> 
> Jails:  1904655
> total:  1955516
> 
> I read somewhere that the virtual memory system pre-pages modified pages as a just-in-case measure. If this is correct, 12.1 does not do this on a non-paging system. The system shown above uses about 10% swapspace after a reboot and works its way to the 42% shown above in a day or so.