Re: What can I learn about data that is staying paged out? (There is a more specific poudriere bulk related context given.)

On 2022-Jun-5, at 12:42, Mark Millard <marklmi@yahoo.com> wrote:

> I have a poudriere bulk -a -c going on a 8 Gibyte
> aarch64 system. top has been showing an occasionally
> increasing swap usage but never any sizable decreases.
> Over 5800 ports have built so far. The context is UFS
> only. The system is running a non-debug build of main.
> 
> Part of the context is ( in /etc/sysctl.conf ):
> 
> vm.swap_enabled=0
> vm.swap_idle_enabled=0
> 
> Also ( in /usr/local/etc/poudriere.conf ):
> 
> USE_TMPFS="data"
> 
> poudriere's TMPFS reports normally total under 128
> KiBytes across the 4 builders.
> 
> For reference, example figures . . .
> 
> A top variant shows:
> 
> Swap: 30720Mi Total, 306816Ki Used
> 
> vmstat -s shows:
> 
>    78152 swap pager pages paged out
> 
> Note: (78152*4096)/1024 == 312608Ki
> 
> So nearly all of the "swap pager pages paged out"
> pages are still sitting out in the used swap/paging
> space. Thus, the usage is not held by user processes
> or is held via very long running processes or is
> not directly tied to user processes --or some mix.
> 
> The variant of top reports never having observed
> more than: 6658Mi MaxObs(Act+Wir+Lndry).
> ("MaxObs" is short for "Maximum Observed".)
> Such high usage is for a bounded time, long past
> at this point. (Until some combination of port
> builds ends up active that uses such.)
> 
> So I'm curious:
> 
> What can I learn about the data that is staying
> paged out (and is gradually growing)? How can I
> learn it?
> 
> 
> Other notes:
> 
> The poudriere jail being built is:
> 
> # poudriere jail -jmain-CA7-bulk_a -i
> Jail name:         main-CA7-bulk_a
> Jail version:      14.0-CURRENT
> Jail arch:         arm.armv7
> Jail method:       null
> Jail mount:        /usr/obj/DESTDIRs/main-CA7-poud-bulk_a
> Jail fs:           
> Jail updated:      2022-05-23 02:21:24
> Jail pkgbase:      disabled
> 
> (Just in case the armv7 jail usage or the null method
> or such is important to the issue.)

Hmm. systat -swap reports a toal for the Devices/Paths Used
that is somewhat less than the total for what reports for the
Pid . . . Total figures (not the Pid Swap figures!):

# systat -swap
                    /0   /1   /2   /3   /4   /5   /6   /7   /8   /9   /10
     Load Average   ||||||||   

Device/Path       Size  Used |0%  /10  /20  /30  /40  / 60\  70\  80\  90\ 100|
gpt/CA72USBswp14   14G  150M
gpt/CA72USBswp16   16G  150M
Total              30G  300M

Pid    Username   Command     Swap/Total Per-Process    Per-System
  1453 root       nfsd         1M /  15M  9%              0%
  1451 root       mountd       1M /  15M  7%              0%
  1481 root       sshd       912K /  20M  4%              0%
  1406 root       ntpd       740K /  27M  2%              0%
  1513 root       login      724K /  14M  5%              0%
  1514 root       sh         656K /  13M  4%              0%
   342 _dhcp      dhclient   516K /  13M  3%              0%
  1363 root       rpcbind    448K /  13M  3%              0%
  1454 root       nfsd       400K /  12M  3%              0%
   341 root       dhclient   380K /  13M  2%              0%
  1341 root       syslogd    324K /  12M  2%              0%
  1505 root       getty      292K /  12M  2%              0%
  1510 root       getty      292K /  12M  2%              0%
  1511 root       getty      292K /  12M  2%              0%
  1512 root       getty      292K /  12M  2%              0%
  1509 root       getty      292K /  12M  2%              0%
  1508 root       getty      292K /  12M  2%              0%
  1507 root       getty      292K /  12M  2%              0%
  1506 root       getty      288K /  12M  2%              0%
  1135 root       devd       272K /  11M  2%              0%
   338 root       dhclient   264K /  13M  2%              0%
     1 root       init       244K /  11M  2%              0%
  1486 root       cron       188K /  13M  1%              0%

I'm, Still looking for a clear indication of what
most of the 300 MiBytes or so of swap/paging space
is in use for.

===
Mark Millard
marklmi at yahoo.com