Re: What can I learn about data that is staying paged out? (There is a more specific poudriere bulk related context given.)

On 2022-Jun-10, at 21:27, Mark Millard <marklmi@yahoo.com> wrote:

> On 2022-Jun-5, at 15:04, Mark Millard <marklmi@yahoo.com> wrote:
> 
>> On 2022-Jun-5, at 12:42, Mark Millard <marklmi@yahoo.com> wrote:
>> 
>>> I have a poudriere bulk -a -c going on a 8 Gibyte
>>> aarch64 system. top has been showing an occasionally
>>> increasing swap usage but never any sizable decreases.
>>> Over 5800 ports have built so far. The context is UFS
>>> only. The system is running a non-debug build of main.
>>> 
>>> Part of the context is ( in /etc/sysctl.conf ):
>>> 
>>> vm.swap_enabled=0
>>> vm.swap_idle_enabled=0
>>> 
>>> Also ( in /usr/local/etc/poudriere.conf ):
>>> 
>>> USE_TMPFS="data"
>>> 
>>> poudriere's TMPFS reports normally total under 128
>>> KiBytes across the 4 builders.
>>> 
>>> For reference, example figures . . .
>>> 
>>> A top variant shows:
>>> 
>>> Swap: 30720Mi Total, 306816Ki Used
>>> 
>>> vmstat -s shows:
>>> 
>>>  78152 swap pager pages paged out
>>> 
>>> Note: (78152*4096)/1024 == 312608Ki
>>> 
>>> So nearly all of the "swap pager pages paged out"
>>> pages are still sitting out in the used swap/paging
>>> space. Thus, the usage is not held by user processes
>>> or is held via very long running processes or is
>>> not directly tied to user processes --or some mix.
>>> 
>>> The variant of top reports never having observed
>>> more than: 6658Mi MaxObs(Act+Wir+Lndry).
>>> ("MaxObs" is short for "Maximum Observed".)
>>> Such high usage is for a bounded time, long past
>>> at this point. (Until some combination of port
>>> builds ends up active that uses such.)
>>> 
>>> So I'm curious:
>>> 
>>> What can I learn about the data that is staying
>>> paged out (and is gradually growing)? How can I
>>> learn it?
>>> 
>>> 
>>> Other notes:
>>> 
>>> The poudriere jail being built is:
>>> 
>>> # poudriere jail -jmain-CA7-bulk_a -i
>>> Jail name:         main-CA7-bulk_a
>>> Jail version:      14.0-CURRENT
>>> Jail arch:         arm.armv7
>>> Jail method:       null
>>> Jail mount:        /usr/obj/DESTDIRs/main-CA7-poud-bulk_a
>>> Jail fs:           
>>> Jail updated:      2022-05-23 02:21:24
>>> Jail pkgbase:      disabled
>>> 
>>> (Just in case the armv7 jail usage or the null method
>>> or such is important to the issue.)
>> 
>> Hmm. systat -swap reports a toal for the Devices/Paths Used
>> that is somewhat less than the total for what reports for the
>> Pid . . . Total figures (not the Pid Swap figures!):
>> 
>> # systat -swap
>>                   /0   /1   /2   /3   /4   /5   /6   /7   /8   /9   /10
>>    Load Average   ||||||||   
>> 
>> Device/Path       Size  Used |0%  /10  /20  /30  /40  / 60\  70\  80\  90\ 100|
>> gpt/CA72USBswp14   14G  150M
>> gpt/CA72USBswp16   16G  150M
>> Total              30G  300M
>> 
>> Pid    Username   Command     Swap/Total Per-Process    Per-System
>> 1453 root       nfsd         1M /  15M  9%              0%
>> 1451 root       mountd       1M /  15M  7%              0%
>> 1481 root       sshd       912K /  20M  4%              0%
>> 1406 root       ntpd       740K /  27M  2%              0%
>> 1513 root       login      724K /  14M  5%              0%
>> 1514 root       sh         656K /  13M  4%              0%
>>  342 _dhcp      dhclient   516K /  13M  3%              0%
>> 1363 root       rpcbind    448K /  13M  3%              0%
>> 1454 root       nfsd       400K /  12M  3%              0%
>>  341 root       dhclient   380K /  13M  2%              0%
>> 1341 root       syslogd    324K /  12M  2%              0%
>> 1505 root       getty      292K /  12M  2%              0%
>> 1510 root       getty      292K /  12M  2%              0%
>> 1511 root       getty      292K /  12M  2%              0%
>> 1512 root       getty      292K /  12M  2%              0%
>> 1509 root       getty      292K /  12M  2%              0%
>> 1508 root       getty      292K /  12M  2%              0%
>> 1507 root       getty      292K /  12M  2%              0%
>> 1506 root       getty      288K /  12M  2%              0%
>> 1135 root       devd       272K /  11M  2%              0%
>>  338 root       dhclient   264K /  13M  2%              0%
>>    1 root       init       244K /  11M  2%              0%
>> 1486 root       cron       188K /  13M  1%              0%
>> 
>> I'm, Still looking for a clear indication of what
>> most of the 300 MiBytes or so of swap/paging space
>> is in use for.
> 
> I finally gave up and checked if a swapoff would
> actually bring in all the pages from swap space
> that were needed (if any) and then un-configure
> the swap space. It did. (The bulk -a was still
> ongoing. It was not doing memory-hog builder
> activity at the time.)
> 
> So such an activity may be a workaround for long
> running things like bulk -a to avoid a swap space
> accumulation that seems to be happening.
> 
> I do not know how much was brought in to RAM vs.
> simply deallocated from swap space (pages not
> changed and still in RAM). If I do such a test
> again, it would be good to figure out how to
> monitor what the swapoff does for bringing in
> pages vs. just discarding them --if possible.
> 
> After a while 12136Ki Used showed up after the
> swapon that reconfigured the swap space, which is
> about the size of the increments that I'd observed
> for its sustained increases.
> 

An interesting point for "systat -swap" now:

                    /0   /1   /2   /3   /4   /5   /6   /7   /8   /9   /10
     Load Average   |||||||||||||||||||||||||||||

Device/Path       Size  Used |0%  /10  /20  /30  /40  / 60\  70\  80\  90\ 100|
gpt/CA72USBswp14   14G 6108K
gpt/CA72USBswp16   16G 6028K
Total              30G   12M

Pid    Username   Command     Swap/Total Per-Process    Per-System


No process is listed as using swap but the 12M
shows as used! That should be a hint, not that
it is directly useful for me figuring out what
the usage is from/for.


===
Mark Millard
marklmi at yahoo.com