Re: Troubles building world on stable/13 [the little bit of evidence about the compiler failures: a jemalloc-tie/ASLR-tie?]
Date: Mon, 07 Feb 2022 23:16:31 UTC
The primary evidence that I have gotten is dumps of
part of the stack spanning the failure in the
stable/13 c++ compiler. [Bob does his own buildworld
buildkernel activities on the RPi3* in question, but
normally absent the likes of make.conf and src.conf
(or equivalents).]
The following did lead to such a stack dump
showing 8 bytes of 0xa5 that otherwise was 0x00.
[The "junk:true" is a means of having jemalloc
fill allocated memory with 0xa5 on allocation
and 0x5a on deallocation (when jemalloc is built
to allow such).]
# ls -Tld /etc/malloc.conf
lrwxr-xr-x 1 root wheel 20 Feb 4 03:47:13 2022 /etc/malloc.conf -> junk:true,abort:true
It was around the beginning of the region that
looked to have been stomped on:
0xffffffffa360: 00 00 00 00 00 00 00 00 a5 a5 a5 a5 a5 a5 a5 a5 ................
Elsewhere in the dumped subregion of the
stack (smaller addresses), there was also an
example of "a5 a5 a5" :
0xffffffffae20: b8 3f 53 00 00 00 00 00 02 22 71 01 c1 a5 a5 a5 .?S......"q.....
There were no examples of "5a 5a" in the region and
only the above examples of back to back a5's. (I did
not check for back-to-back across lines.)
What looks to be a valid fp/lr pair is:
0xffffffffa2d0: 70 a5 ff ff ff ff 00 00 6c 2b b7 02 00 00 00 00 p.......l+......
But in the dump what the left part (fp part)
refers to is:
0xffffffffa570: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
For reference for lr's "6c 2b b7 02":
(lldb) disass -c 10 -a 0x2b72b6c
c++`::ParseOptionalCXXScopeSpecifier():
0x2b723fc <+0>: stp x29, x30, [sp, #-0x60]!
0x2b72400 <+4>: stp x28, x27, [sp, #0x10]
0x2b72404 <+8>: stp x26, x25, [sp, #0x20]
0x2b72408 <+12>: stp x24, x23, [sp, #0x30]
0x2b7240c <+16>: stp x22, x21, [sp, #0x40]
0x2b72410 <+20>: stp x20, x19, [sp, #0x50]
0x2b72414 <+24>: mov x29, sp
0x2b72418 <+28>: sub sp, sp, #0x250 ; =0x250
0x2b7241c <+32>: adrp x8, 9734
0x2b72420 <+36>: ldr x8, [x8, #0xe60]
Most failures have the 0x01 after the ": ", but
on occasion I've gotten one with 0x05 instead.
The surrounding lines for the example at hand,
that follow a simple, similarity-pattern, look
like:
0xffffffffa450: 01 00 00 00 00 00 00 00 80 32 b2 55 00 00 00 00 .........2.U....
0xffffffffa460: 00 00 00 00 00 00 00 00 e2 34 b2 55 00 00 00 00 .........4.U....
0xffffffffa470: 01 00 00 00 00 00 00 00 00 33 b2 55 00 00 00 00 .........3.U....
0xffffffffa480: 00 00 00 00 00 00 00 00 f2 34 b2 55 00 00 00 00 .........4.U....
0xffffffffa490: 01 00 00 00 00 00 00 00 80 33 b2 55 00 00 00 00 .........3.U....
0xffffffffa4a0: 00 00 00 00 00 00 00 00 02 35 b2 55 00 00 00 00 .........5.U....
0xffffffffa4b0: 01 00 00 00 00 00 00 00 00 34 b2 55 00 00 00 00 .........4.U....
0xffffffffa4c0: 00 00 00 00 00 00 00 00 12 35 b2 55 00 00 00 00 .........5.U....
0xffffffffa4d0: 01 00 00 00 00 00 00 00 80 34 b2 55 00 00 00 00 .........4.U....
0xffffffffa4e0: 00 00 00 00 00 00 00 00 22 35 b2 55 00 00 00 00 ........"5.U....
0xffffffffa4f0: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa500: 00 00 00 00 00 00 00 00 5a 35 b2 55 00 00 00 00 ........Z5.U....
0xffffffffa510: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa520: 00 00 00 00 00 00 00 00 a2 35 b2 55 00 00 00 00 .........5.U....
0xffffffffa530: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa540: 00 00 00 00 00 00 00 00 ea 35 b2 55 00 00 00 00 .........5.U....
0xffffffffa550: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa560: 00 00 00 00 00 00 00 00 32 36 b2 55 00 00 00 00 ........26.U....
0xffffffffa570: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa580: 00 00 00 00 00 00 00 00 7a 36 b2 55 00 00 00 00 ........z6.U....
0xffffffffa590: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa5a0: 00 00 00 00 00 00 00 00 c2 36 b2 55 00 00 00 00 .........6.U....
0xffffffffa5b0: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa5c0: 00 00 00 00 00 00 00 00 0a 37 b2 55 00 00 00 00 .........7.U....
0xffffffffa5d0: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa5e0: 00 00 00 00 00 00 00 00 52 37 b2 55 00 00 00 00 ........R7.U....
0xffffffffa5f0: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa600: 00 00 00 00 00 00 00 00 9a 37 b2 55 00 00 00 00 .........7.U....
0xffffffffa610: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa620: 00 00 00 00 00 00 00 00 e2 37 b2 55 00 00 00 00 .........7.U....
0xffffffffa630: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa640: 00 00 00 00 00 00 00 00 2a 38 b2 55 00 00 00 00 ........*8.U....
0xffffffffa650: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa660: 00 00 00 00 00 00 00 00 72 38 b2 55 00 00 00 00 ........r8.U....
0xffffffffa670: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa680: 00 00 00 00 00 00 00 00 ba 38 b2 55 00 00 00 00 .........8.U....
0xffffffffa690: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa6a0: 00 00 00 00 00 00 00 00 02 39 b2 55 00 00 00 00 .........9.U....
0xffffffffa6b0: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa6c0: 00 00 00 00 00 00 00 00 4a 39 b2 55 00 00 00 00 ........J9.U....
0xffffffffa6d0: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa6e0: 00 00 00 00 00 00 00 00 92 39 b2 55 00 00 00 00 .........9.U....
0xffffffffa6f0: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa700: 00 00 00 00 00 00 00 00 da 39 b2 55 00 00 00 00 .........9.U....
0xffffffffa710: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa720: 00 00 00 00 00 00 00 00 22 3a b2 55 00 00 00 00 ........":.U....
0xffffffffa730: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa740: 00 00 00 00 00 00 00 00 6a 3a b2 55 00 00 00 00 ........j:.U....
0xffffffffa750: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa760: 00 00 00 00 00 00 00 00 b2 3a b2 55 00 00 00 00 .........:.U....
0xffffffffa770: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa780: 00 00 00 00 00 00 00 00 fa 3a b2 55 00 00 00 00 .........:.U....
0xffffffffa790: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa7a0: 00 00 00 00 00 00 00 00 42 3b b2 55 00 00 00 00 ........B;.U....
0xffffffffa7b0: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa7c0: 00 00 00 00 00 00 00 00 8a 3b b2 55 00 00 00 00 .........;.U....
0xffffffffa7d0: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa7e0: 00 00 00 00 00 00 00 00 d2 3b b2 55 00 00 00 00 .........;.U....
0xffffffffa7f0: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa800: 00 00 00 00 00 00 00 00 1a 3c b2 55 00 00 00 00 .........<.U....
0xffffffffa810: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa820: 00 00 00 00 00 00 00 00 62 3c b2 55 00 00 00 00 ........b<.U....
0xffffffffa830: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa840: 00 00 00 00 00 00 00 00 aa 3c b2 55 00 00 00 00 .........<.U....
0xffffffffa850: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa860: 00 00 00 00 00 00 00 00 f2 3c b2 55 00 00 00 00 .........<.U....
0xffffffffa870: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa880: 00 00 00 00 00 00 00 00 3a 3d b2 55 00 00 00 00 ........:=.U....
0xffffffffa890: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa8a0: 00 00 00 00 00 00 00 00 82 3d b2 55 00 00 00 00 .........=.U....
0xffffffffa8b0: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa8c0: 00 00 00 00 00 00 00 00 ca 3d b2 55 00 00 00 00 .........=.U....
0xffffffffa8d0: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa8e0: 00 00 00 00 00 00 00 00 12 3e b2 55 00 00 00 00 .........>.U....
0xffffffffa8f0: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa900: 00 00 00 00 00 00 00 00 5a 3e b2 55 00 00 00 00 ........Z>.U....
0xffffffffa910: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa920: 00 00 00 00 00 00 00 00 a2 3e b2 55 00 00 00 00 .........>.U....
0xffffffffa930: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa940: 00 00 00 00 00 00 00 00 ea 3e b2 55 00 00 00 00 .........>.U....
0xffffffffa950: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa960: 00 00 00 00 00 00 00 00 32 3f b2 55 00 00 00 00 ........2?.U....
0xffffffffa970: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa980: 00 00 00 00 00 00 00 00 7a 3f b2 55 00 00 00 00 ........z?.U....
0xffffffffa990: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa9a0: 00 00 00 00 00 00 00 00 c2 3f b2 55 00 00 00 00 .........?.U....
0xffffffffa9b0: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa9c0: 00 00 00 00 00 00 00 00 0a 40 b2 55 00 00 00 00 .........@.U....
0xffffffffa9d0: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffa9e0: 00 00 00 00 00 00 00 00 52 40 b2 55 00 00 00 00 ........R@.U....
0xffffffffa9f0: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffaa00: 00 00 00 00 00 00 00 00 9a 40 b2 55 00 00 00 00 .........@.U....
0xffffffffaa10: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffaa20: 00 00 00 00 00 00 00 00 e2 40 b2 55 00 00 00 00 .........@.U....
0xffffffffaa30: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffaa40: 00 00 00 00 00 00 00 00 2a 41 b2 55 00 00 00 00 ........*A.U....
0xffffffffaa50: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffaa60: 00 00 00 00 00 00 00 00 72 41 b2 55 00 00 00 00 ........rA.U....
0xffffffffaa70: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 ...........U....
0xffffffffaa80: 00 00 00 00 00 00 00 00 ba 41 b2 55 00 00 00 00 .........A.U....
When the 0x05's show up they are instead of the
0x01's, just after the ": ".
After that the pattern is different. But quickly
something looks like another fp/lr pair in memory,
and tha, in turn, it references another:
0xffffffffaa90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0xffffffffaaa0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0xffffffffaab0: 00 00 00 00 00 00 00 00 44 c4 95 07 0e 02 46 57 ........D.....FW
0xffffffffaac0: 10 ab ff ff ff ff 00 00 8c c6 aa 02 00 00 00 00 ................
. . .
0xffffffffab10: 90 ac ff ff ff ff 00 00 e0 18 ab 02 00 00 00 00 ................
. . .
But after that the following does not seem to
fit the pattern:
0xffffffffac90: 00 ac ff ff ff ff 00 00 44 c4 95 07 0e 02 46 57 ........D.....FW
and:
0xffffffffac00: 01 00 00 00 00 00 00 00 18 ae ff ff ff ff 00 00 ................
The a5 sequences make me wonder if jemalloc
assigned a memory allocation to stack space or
was told to handle a stack address as if it was
an assigned address for some aspects of an
allocation (if that can even be requested).
I wonder if there is any chance of ASLR being
involved with the stack and memory allocation
possibly overlapping.
But I've really no clue.
I've given up on trying to isolate what is going
on for the compiler failures. I've only been able
to see after the failure, not just before:
debugger interactions with the compiler process
in times close to the failure point in the code
prevent the failure. I've not found any
alternative that avoids such.
This is on top of the issue that the plain-runs
(no debugger) vary in behavior, sometimes
running to completion, sometimes stopping at
similar but varying places in the source code
being processed. There is still no known way to
get a full reproduction of failure details each
time. (Which instance of the example type of
source code being compiled at the point of
failure does vary.)
For reference: I've been using .sh/.cpp
pairs that Bob published and a copy of the
c++ from his system to investigate. The
.cpp is large. Bob's RPi3* is a RAM+SWAP
context of: 1 GiBYTe + 2 GiByte and I made
such a context on a RPi3* as well. But I
ran his stable/13 c++ on a system with a
non-debug main [so: 14] kernel and either
a main world or a stable/13 chroot. From
the chroot:
# uname -apKU
FreeBSD Rock64_RPi_4_3_2v1p2 14.0-CURRENT FreeBSD 14.0-CURRENT #28 main-n252475-e76c0108990b-dirty: Sat Jan 15 23:39:27 PST 2022 root@CA72_16Gp_ZFS:/usr/obj/BUILDs/main-CA53-nodbg-clang/usr/main-src/arm64.aarch64/sys/GENERIC-NODBG-CA53 arm64 aarch64 1400047 1300524
# freebsd-version -ru
14.0-CURRENT
13.0-STABLE
# ~/fbsd-based-on-what-commit.sh -C /usr/13S-src/
branch: stable/13
merge-base: a5f69859956049b5153b0e1b67f8f4a99622dc6f
merge-base: CommitDate: 2022-01-15 12:55:32 +0000
a5f698599560 (HEAD -> stable/13, freebsd/stable/13) Ignore debugger-injected signals left after detaching
Bob's recent stable/13 context (kernel too) is
more recent than mine. So the problems has been
observed over a range of contexts.
But, as I said, I've given up on finding a
way to isolate whatever is going on.
===
Mark Millard
marklmi at yahoo.com