Re: What can I learn about data that is staying paged out? (There is a more specific poudriere bulk related context given.)

On 2022-Jun-5, at 15:04, Mark Millard <marklmi@yahoo.com> wrote:

> On 2022-Jun-5, at 12:42, Mark Millard <marklmi@yahoo.com> wrote:
> 
>> I have a poudriere bulk -a -c going on a 8 Gibyte
>> aarch64 system. top has been showing an occasionally
>> increasing swap usage but never any sizable decreases.
>> Over 5800 ports have built so far. The context is UFS
>> only. The system is running a non-debug build of main.
>> 
>> Part of the context is ( in /etc/sysctl.conf ):
>> 
>> vm.swap_enabled=0
>> vm.swap_idle_enabled=0
>> 
>> Also ( in /usr/local/etc/poudriere.conf ):
>> 
>> USE_TMPFS="data"
>> 
>> poudriere's TMPFS reports normally total under 128
>> KiBytes across the 4 builders.
>> 
>> For reference, example figures . . .
>> 
>> A top variant shows:
>> 
>> Swap: 30720Mi Total, 306816Ki Used
>> 
>> vmstat -s shows:
>> 
>>   78152 swap pager pages paged out
>> 
>> Note: (78152*4096)/1024 == 312608Ki
>> 
>> So nearly all of the "swap pager pages paged out"
>> pages are still sitting out in the used swap/paging
>> space. Thus, the usage is not held by user processes
>> or is held via very long running processes or is
>> not directly tied to user processes --or some mix.
>> 
>> The variant of top reports never having observed
>> more than: 6658Mi MaxObs(Act+Wir+Lndry).
>> ("MaxObs" is short for "Maximum Observed".)
>> Such high usage is for a bounded time, long past
>> at this point. (Until some combination of port
>> builds ends up active that uses such.)
>> 
>> So I'm curious:
>> 
>> What can I learn about the data that is staying
>> paged out (and is gradually growing)? How can I
>> learn it?
>> 
>> 
>> Other notes:
>> 
>> The poudriere jail being built is:
>> 
>> # poudriere jail -jmain-CA7-bulk_a -i
>> Jail name:         main-CA7-bulk_a
>> Jail version:      14.0-CURRENT
>> Jail arch:         arm.armv7
>> Jail method:       null
>> Jail mount:        /usr/obj/DESTDIRs/main-CA7-poud-bulk_a
>> Jail fs:           
>> Jail updated:      2022-05-23 02:21:24
>> Jail pkgbase:      disabled
>> 
>> (Just in case the armv7 jail usage or the null method
>> or such is important to the issue.)
> 
> Hmm. systat -swap reports a toal for the Devices/Paths Used
> that is somewhat less than the total for what reports for the
> Pid . . . Total figures (not the Pid Swap figures!):
> 
> # systat -swap
>                    /0   /1   /2   /3   /4   /5   /6   /7   /8   /9   /10
>     Load Average   ||||||||   
> 
> Device/Path       Size  Used |0%  /10  /20  /30  /40  / 60\  70\  80\  90\ 100|
> gpt/CA72USBswp14   14G  150M
> gpt/CA72USBswp16   16G  150M
> Total              30G  300M
> 
> Pid    Username   Command     Swap/Total Per-Process    Per-System
>  1453 root       nfsd         1M /  15M  9%              0%
>  1451 root       mountd       1M /  15M  7%              0%
>  1481 root       sshd       912K /  20M  4%              0%
>  1406 root       ntpd       740K /  27M  2%              0%
>  1513 root       login      724K /  14M  5%              0%
>  1514 root       sh         656K /  13M  4%              0%
>   342 _dhcp      dhclient   516K /  13M  3%              0%
>  1363 root       rpcbind    448K /  13M  3%              0%
>  1454 root       nfsd       400K /  12M  3%              0%
>   341 root       dhclient   380K /  13M  2%              0%
>  1341 root       syslogd    324K /  12M  2%              0%
>  1505 root       getty      292K /  12M  2%              0%
>  1510 root       getty      292K /  12M  2%              0%
>  1511 root       getty      292K /  12M  2%              0%
>  1512 root       getty      292K /  12M  2%              0%
>  1509 root       getty      292K /  12M  2%              0%
>  1508 root       getty      292K /  12M  2%              0%
>  1507 root       getty      292K /  12M  2%              0%
>  1506 root       getty      288K /  12M  2%              0%
>  1135 root       devd       272K /  11M  2%              0%
>   338 root       dhclient   264K /  13M  2%              0%
>     1 root       init       244K /  11M  2%              0%
>  1486 root       cron       188K /  13M  1%              0%
> 
> I'm, Still looking for a clear indication of what
> most of the 300 MiBytes or so of swap/paging space
> is in use for.

I finally gave up and checked if a swapoff would
actually bring in all the pages from swap space
that were needed (if any) and then un-configure
the swap space. It did. (The bulk -a was still
ongoing. It was not doing memory-hog builder
activity at the time.)

So such an activity may be a workaround for long
running things like bulk -a to avoid a swap space
accumulation that seems to be happening.

I do not know how much was brought in to RAM vs.
simply deallocated from swap space (pages not
changed and still in RAM). If I do such a test
again, it would be good to figure out how to
monitor what the swapoff does for bringing in
pages vs. just discarding them --if possible.

After a while 12136Ki Used showed up after the
swapon that reconfigured the swap space, which is
about the size of the increments that I'd observed
for its sustained increases.


===
Mark Millard
marklmi at yahoo.com