rpi2 hangup during poudriere build: lots of pfault wmseg status

Thu Dec 7 05:00:20 UTC 2017

> On 2017-Dec-6, at 5:47 PM, Laurent Cimon <laurent at nuxi.ca> wrote:
> 
>> On Dec 6, 2017, at 20:01, Mark Millard <markmi at dsl-only.net> wrote:
>> 
>> On 2017-Dec-6, at 1:54 PM, Laurent Cimon <laurent at nuxi.ca> wrote:
>> 
>>>> On Dec 6, 2017, at 00:57, Mark Millard <markmi at dsl-only.net> wrote:
>>>> 
>>>> I tried to build some ports on a rpi2
>>>> (via poudriere) but it hung up:
>>>> Ethernet and normal console use. (Note:
>>>> the root file system is on a USB SSD
>>>> and the swap partition is also on that
>>>> USB SSD.)
>>>> 
>>>> But ~^b worked for getting to the db>
>>>> prompt on the console.
>>>> 
>>>> From there a ps suggests that it got hung
>>>> up in pfault activity. (Possibly insufficient
>>>> RAM+swap-partition space?) But it is not
>>>> clear to me that it should end up hung up
>>>> vs. killing processes or other such.
>>> 
>>> Hi,
>>> 
>>> From what I know the raspberry pis use the same controller for ethernet and
>>> the USB hub on which you’re hosting an SSD. It seems like you make very heavy
>>> use of the USB ports, and all of the resources used by poudriere except for the
>>> CPU and the (very limited) memory that’s not in swap is attached to them. If you
>>> really didn’t have enough memory and swap, the linkers would’ve been stopped.
>>> 
>>> I think it might just be a swap death. Poudriere compiles and fetches in parallel
>>> a lot, ethernet and disk I/O is slow because it’s very limited, so linking takes
>>> longer. You end up linking a few very big binaries at the same time, and they
>>> all fight for the memory, to get out of swap through page faults, but there
>>> are too many page faults, all too big, requesting for more CPU time that’s
>>> allowed to them.
>>> 
>>> This would explain why you have 3 linkers waiting on a page fault out of the 4
>>> CPUs poudriere allows builds on, on top of the awk processes. It would also
>>> explain why you had easy access to the debugger: it was in memory already with
>>> the kernel.
>>> 
>>> I’d advise you to disable parallel builds and see if it happens again,
>>> but it would make building much slower. Using makejobs would help if you
>>> can afford watching the build. Otherwise be patient, it should resolve itself
>>> eventually, but it will take a while and it will happen again.
>> 
>> My post was more about how FreeBSD handled the
>> heavy-use context and less about getting the
>> builds to finish: it managed to to get to a
>> state of no-progress for processes and a loss
>> of normal control as far as I could tell.
>> 
>> I did a "c" to ddb and left it until just before
>> this note then did ~ ^B again. Things looked the
>> same. [I've finally rebooted the rpi2.]
>> 
>> PARALLEL_JOBS=1 was already in use but
>> ALLOW_MAKE_JOBS=yes was also in use.
>> USE_TMPFS=no was already in use.
>> 
>> While an ssh session was monitoring the
>> build, Ethernet was not in heavy use.
>> (No nfs mounts to its disks, for example.)
>> 
>> I may try without ALLOW_MAKE_JOBS=yes and
>> with ALLOW_MAKE_JOBS_PACKAGES empty/undefined
>> to see if it can complete for such a context
>> without having the same sort of problem.
>> 
>> Ultimately I can cross-build and install from
>> those materials when I really want updates. I
>> have the context for such. This was more about
>> seeing how well the rpi2 did for self-hosted.
>> Classically I've used a BPI-M3 with 2 GiBytes
>> of RAM and a proportionally bigger swap partition
>> instead (approximately).
>> 
>> 
>> FYI (rpi2 after rebooting):
>> 
>> # swapinfo
>> Device          1K-blocks     Used    Avail Capacity
>> /dev/label/RPI2swap   1572860        0  1572860     0%
>> 
>> # df -m
>> Filesystem           1M-blocks  Used  Avail Capacity  Mounted on
>> /dev/ufs/RPI2rootfs     195378 30791 148957    17%    /
>> devfs                        0     0      0   100%    /dev
>> /dev/label/RPI2Aboot        49    12     37    25%    /boot/msdos
>> 
>> 
>> An rpi3 (aarch64) with the same amount of RAM,
>> same type of USB SSD, etc., but well more swap
>> completed building basically the same set of
>> ports for the same poudriere settings just
>> fine.
>> 
>> Interestingly for the default kern.maxswzone:
>> (Just to show the reported recommended maximum
>> figures for swap.)
>> 
>> rpi2: . . . exceeds maximum recommended amount (411488 pages).
>> rpi3: . . . exceeds maximum recommended amount (925680 pages).
>> 
>> (I was running with somewhat under those maximums for
>> the tests.)
>> 
>> # swapinfo
>> Device          1K-blocks     Used    Avail Capacity
>> /dev/gpt/RPI3swap   3702784        0  3702784     0%
>> 
>> # df -m
>> Filesystem           1M-blocks  Used  Avail Capacity  Mounted on
>> /dev/ufs/RPI3rootfs     195378 14937 164811     8%    /
>> devfs                        0     0      0   100%    /dev
>> /dev/label/RPI3Aboot        49     7     42    15%    /boot/efi
>> 
>> If I restricted the rpi3 to somewhat under what the
>> rpi2 allows for swap, I do not know if it would also
>> hang up vs. not.
>> 
>> If having more swap makes the difference, then it
>> would not seem to be being I/O-bound that would
>> explain the hangup.
>> 
>> 
>> ===
>> Mark Millard
>> markmi at dsl-only.net
> 
> There are a few factors that could have prevented this on your raspberry pi 3.
> 
> It has a faster, 64 bit CPU instead of the raspberry pi 2’s 32 bit CPU and the
> RAM is twice as fast. These make it less likely for this to happen, because it
> makes both building and linking faster, which reduces the odds of linking 2
> binaries at once, let alone 3. There are many things that could have gone
> differently in the build that didn’t make it end up linking 3 big binaries at
> the same time to cause the same behaviour.
> 
> What I think happened on your raspberry pi 2 is just likely bad luck that could
> also happen on your raspberry pi 3. The odds of 3 parallel builds needing so
> much ram to link at the exact same time are still very low, just less low on
> faster hardware.
> 
> Keep in mind that this is still entirely theoretical, I don’t present it as an
> absolute explanation. It’s simply what I understand from this.
> 
> I’d be curious seeing how a different operating system using a system similar to
> poudriere where builds are done on one CPU but in parallel would be handled on
> the rpi2. My understanding is that this is simply a mix of hardware limitation
> and conceptual flaws with the swap. And by flaws I mean, your operating system
> cannot save you when you try to do something that your hardware cannot possibly
> do.

For reference:

The rpi2 hung up during:

[08:00:15] [01] [00:00:00] Building devel/binutils | binutils-2.28,1

(Only one builder, no prior builds should matter. All 4 cores
allowed.)

On the rpi3 this was:

[08:13:38] [01] [00:00:00] Building devel/binutils | binutils-2.28,1
[10:17:12] [01] [02:03:34] Finished devel/binutils | binutils-2.28,1: Success

(Only one builder, no prior or following builds
should matter. All 4 cores allowed.)

Comparing a couple of examples that both completed:

rpi2:
[00:43:40] [01] [00:00:00] Building lang/perl5.24 | perl5-5.24.3
[01:38:37] [01] [00:54:57] Finished lang/perl5.24 | perl5-5.24.3: Success
vs.
rpi3:
[00:26:35] [01] [00:00:00] Building lang/perl5.24 | perl5-5.24.3
[00:56:14] [01] [00:29:39] Finished lang/perl5.24 | perl5-5.24.3: Success

rpi2:
[07:12:51] [01] [00:00:00] Building databases/sqlite3 | sqlite3-3.21.0_1
[07:59:04] [01] [00:46:13] Finished databases/sqlite3 | sqlite3-3.21.0_1: Success
vs.
rpi3:
[07:43:31] [01] [00:00:00] Building databases/sqlite3 | sqlite3-3.21.0_1
[08:13:35] [01] [00:30:04] Finished databases/sqlite3 | sqlite3-3.21.0_1: Success

The rpi2 lasting days longer than the rpi3 2hr
figure for devel/binutils is likely out of scale
for processor and RAM differences in speed. (The
USB-tied performance likely is not all that
different.)

===
Mark Millard
markmi at dsl-only.net