OOMA kill with vm.pfault_oom_attempts="-1" on RPi3 at r357147

[bob prohaska]

The latest attempt at buildworld on a Pi3 with kernel and sources at
r357147 stopped with an "out of swap" kill. The activity log reported,
in the one second samples before, during and after the kill recorded:

procs     memory       page                      disks     faults       cpu
r b w     avm     fre  flt  re  pi  po    fr   sr mm0 da0   in   sy   cs us sy id
 4  0  0 1670108   58012  1332   5   2   1  1414  597   0   0 10572  1251  2340 80 18  2
dT: 1.015s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s   kBps   ms/d   %busy Name
Mon Jan 27 10:32:06 PST 2020
Device          1K-blocks     Used    Avail Capacity
/dev/mmcsd0s2b    4404252   135540  4268712     3%
/dev/da0p6        5242880   132148  5110732     3%
Total             9647132   267688  9379444     3%
Jan 26 19:44:00 www sshd[1289]: error: maximum authentication attempts exceeded for invalid user  from 45.136.108.85 port 16543 ssh2 [preauth]
Jan 26 19:44:12 www sshd[1298]: error: maximum authentication attempts exceeded for invalid user  from 45.136.108.85 port 4581 ssh2 [preauth]
0/254/254/19180 mbuf clusters in use (current/cache/total/max)

procs     memory       page                      disks     faults       cpu
r b w     avm     fre  flt  re  pi  po    fr   sr mm0 da0   in   sy   cs us sy id
 3  0  0 1683560   54412  1332   5   2   1  1414  603   0   0 10572  1251  2340 80 18  2
dT: 1.051s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s   kBps   ms/d   %busy Name
    0     79      3     11  917.7     76   1557  212.6      0      0    0.0  318.6  mmcsd0
    0     79      3     11  918.0     76   1557  212.8      0      0    0.0  318.6  mmcsd0s2
    0     52      1      4    1.4     51    780    1.4      0      0    0.0    6.9  da0
    0     22      0      0    0.0     22    700  177.2      0      0    0.0   32.5  mmcsd0s2a
    0     57      3     11  918.0     54    856  227.2      0      0    0.0  318.6  mmcsd0s2b
    0     22      0      0    0.0     22    700  177.3      0      0    0.0   32.5  ufs/rootfs
    0     51      1      4    1.4     50    780    1.4      0      0    0.0    7.1  da0p6
Mon Jan 27 10:32:12 PST 2020
Device          1K-blocks     Used    Avail Capacity
/dev/mmcsd0s2b    4404252   137184  4267068     3%
/dev/da0p6        5242880   134284  5108596     3%
Total             9647132   271468  9375664     3%
Jan 26 19:44:00 www sshd[1289]: error: maximum authentication attempts exceeded for invalid user  from 45.136.108.85 port 16543 ssh2 [preauth]
Jan 26 19:44:12 www sshd[1298]: error: maximum authentication attempts exceeded for invalid user  from 45.136.108.85 port 4581 ssh2 [preauth]
0/256/256/19180 mbuf clusters in use (current/cache/total/max)

procs     memory       page                      disks     faults       cpu
r b w     avm     fre  flt  re  pi  po    fr   sr mm0 da0   in   sy   cs us sy id
 3  0  0 1394192  142844  1332   5   2   1  1415  607   0   0 10573  1251  2341 80 18  2
dT: 1.006s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s   kBps   ms/d   %busy Name
    0     25     25    557    1.9      0      0    0.0      0      0    0.0    4.6  mmcsd0
    0     25     25    557    1.9      0      0    0.0      0      0    0.0    4.8  mmcsd0s2
    0     17     17    159    1.3      0      0    0.0      0      0    0.0    2.2  da0
    0      8      8    231    2.4      0      0    0.0      0      0    0.0    1.9  mmcsd0s2a
    0     17     17    326    1.7      0      0    0.0      0      0    0.0    2.9  mmcsd0s2b
    0      8      8    231    2.4      0      0    0.0      0      0    0.0    1.9  ufs/rootfs
    0     17     17    159    1.3      0      0    0.0      0      0    0.0    2.3  da0p6
Mon Jan 27 10:32:21 PST 2020
Device          1K-blocks     Used    Avail Capacity
/dev/mmcsd0s2b    4404252    43020  4361232     1%
/dev/da0p6        5242880    42128  5200752     1%
Total             9647132    85148  9561984     1%
Jan 26 19:44:12 www sshd[1298]: error: maximum authentication attempts exceeded for invalid user  from 45.136.108.85 port 4581 ssh2 [preauth]
Jan 27 10:32:18 www kernel: pid 97756 (c++), jid 0, uid 0, was killed: out of swap space

Here's the command used to collect the activity log:

#!/bin/sh
while true
do vmstat ; gstat -abd -I 1s ; date ; swapinfo ; tail -n 2 /var/log/messages ; netstat -m | grep "mbuf clusters" 
done

It looks as if the vm.pfault_oom_attempts="-1" no longer shuts OOMA off. 
Is there another way to deal with the problem?

As an aside, it appears the activity percentages in top have changed:
Formerly the per-cpu numbers totalled about four times the total %busy.
Now the per-cpu numbers roughly add up to total %busy. Not sure it matters,
but it's certainly different from previous behavior.

Perhaps most surprisingly, after buildworld and the activity logger had
stopped (while I was writing this little missive) the machine again panic'd,
reporting:
panic: deadlres_td_sleep_q: possible deadlock detected for 0xfffffd0000eff000, blocked for 1800269 ticks

Thanks for reading and any ideas.

bob prohaska