Bug 215821 for TARGET_ARCH=powerpc64: bootstrapped ld produces a "shared object" when given a -pie option; junk kernel produced crashes

[Mark Millard]

On 2017-Jan-6, at 7:19 AM, Justin Hibbits <jrh29 at alumni.cwru.edu> wrote:

> On Fri, Jan 6, 2017 at 3:04 AM, Mark Millard <markmi at dsl-only.net> wrote:
>> I have submitted a FreeBSD bugzilla entry:
>> 
>> Bug 215821 - head -r311147's bootstrapped ld for TARGET_ARCH=powerpc64 produces kernel.full as a "shared object" for -pie instead of as a "executable": booting the produced kernel crashes
>> 
>> In essence what the .meta file for kernel.full shows as:
>> 
>> CMD @ld -Bdynamic -T /usr/src/sys/conf/ldscript.powerpc64 -pie --no-warn-mismatch  --warn-common --export-dynamic --dynamic-linker /red/herring  -o kernel.full -X locore.o . . .
>> 
>> (note the -pie) ends up producing:
>> 
>> ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1 (FreeBSD), dynamically linked, interpreter /red/herring, not stripped
>> 
>> instead of what it should have produced:
>> 
>> ELF 64-bit MSB executable, 64-bit PowerPC or cisco 7500, version 1 (FreeBSD), dynamically linked, interpreter /red/herring, not stripped
>> 
>> The differences in content leads to the powerpc64 crashing at the start of the produced kernel.
>> 
>> This means needing to use devel/binutils and/or devel/powerpc64-binutils instead (at least for ld).
>> 
>> ===
>> Mark Millard
>> markmi at dsl-only.net
> 
> Hi Mark,
> 
> Nathan made a change 2 years ago to build the kernel as a shared
> object, so that it can be relocatable.  Looking at my kernel.full
> (base gcc build):
> 
> world/zhabar/home/chmeee/freebsd/pristine/sys/ZHABAR/kernel.full: ELF
> 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1
> (FreeBSD), dynamically linked, interpreter /red/herring, not stripped
> 
> - Justin

A "position independent executable" and a "shared object" are both
relocatable: both are dynamically linked. (The below mentions some
differences that show up.) It looks like Nathan may have tried to
handle what the system ld did with -pie as well as what devel/*binutils
does. (More below.)

Nathon wrote in https://lists.freebsd.org/pipermail/freebsd-ppc/2015-January/007375.html
that:

> This is the first architecture to have a PIE kernel, however, so I'd 
> like some feedback on the approach. The major immediate difficulty is 
> that PIE kernels are ET_DYN ELF executables.

He said "executable", not "shared object" but also said "ET_DYN". I'll
note that the flags from the different versions of ld (system binutils
vs. devel/*binutils) ends up with different flags (system first, then
devel/*binutils) for -pie being specified:

architecture: powerpc:common64, flags 0x00000150:
HAS_SYMS, DYNAMIC, D_PAGED
start address 0x00000000010cf848

The above sort of case also has .branch_lt sections involved.
(gcc 4.2.1 and clang, I've not tried xtoolchain mixed with
the bootstrapped ld.)

architecture: powerpc:common64, flags 0x00000112:
EXEC_P, HAS_SYMS, D_PAGED
start address 0x000000000108b6c8

The above sort of case does not have .branch_lt sections involved.
(clang and xtoolchain, I've not tried gcc 4.2.1 with
devel/*binutils based ld.)

[They are not the same builds so the address
difference is expected. And the start addresses
do not match the "<.__start> mfmsr   r20" in
either case and is not where the kernel code
execution starts on the PowerMac G5's.]

It appears to me that the system ld and the devel/*binutils
ld do not agree for what -pie generates.

I would guess that -pie should involve the EXEC_P flag and
so be a executable according to the flags and that an executable
would not have .branch_lt . In other words:
devel/*binutils is correct in that much by my guess. Another
wording of that would be: for the system ld "-pie" is "in
name only" or has a "special FreeBSD definition" from what I
can tell.

Another difference is that the EXEC_P examples
from devel/*binutils later list:

private flags = 0x1: [abiv1]

at the end of the Dynamic Section but the HAS_SYMS
examples do not list such an indication of the abi.

But that note from Nathan also had:

load_elf.c with:

+#if defined(__powerpc__) && __ELF_WORD_SIZE == 64
+    } else if (ehdr->e_type == ET_EXEC ||
+      (ehdr->e_type == ET_DYN && ehdr->e_entry != 0)) {
+#else
     } else if (ehdr->e_type == ET_EXEC) {
+#endif

Makefile.powerpc with:

+.if ${MACHINE_ARCH} == "powerpc64"
+CFLAGS+= -fPIC
+LDFLAGS+= -pie
+.endif

kmod.mk with:

+# Don't add a fake entry point to modules
+_LDFLAGS+= -e 0

[The .meta files show the -pie option that specifies position
independent executable. That includes when I use gcc 4.2.1
in the modern context.]

load_elf.c suggests trying to handle what the bootstrap ld
does with -pie .

When I try a bootstrapped gcc 4.2.1 based buildkernel I get
a kernel file with:

. . .
Disassembly of section .text:
0000000000100160 <.__start-0x2350> std     r2,40(r1)
0000000000100164 <.__start-0x234c> addis   r2,r2,1
0000000000100168 <.__start-0x2348> addi    r2,r2,-520
000000000010016c <.__start-0x2344> b       0000000000368d38 <.pci_find_dbsf>
0000000000100170 <.__start-0x2340> std     r2,40(r1)
. . .
00000000001024b0 <.__start> mfmsr   r20
00000000001024b4 <.__start+0x4> li      r21,1
00000000001024b8 <.__start+0x8> rldimi  r20,r21,63,0
00000000001024bc <.__start+0xc> mtmsrd  r20
00000000001024c0 <.__start+0x10> isync

Note the "40(r1)" use of an uninitialized r1 if the code
starts at 0000000000100160 and the branch to .pci_find_dbsf
as well. With this code the execution needs to not start
at 0000000000100160 .

[Justin: What does your example look like for such?]

Later I'll try installing and booting this gcc4.2.1 based
kernel but so far it looks like other non-gcc 4.2.1 based
ones that used the bootstrapped ld and that actually start
at what would be 0000000000100160 in the above example above
and quickly fails on the PowerMac G5 so-called "Quad Core"
that I currently have access to.

If it boots I'll see if I can track down another difference
in how things start for kernel execution. But clang mishandles
what should be R_PPC64_TOC16_DS with .toc and so the boot \
fails for lack of correct offsets in addressing, such has
0(r2) instead of something like -32760(r2). This alone blocks
getting very far for booting based on clang: Both variants of
ld get boot failure for clang as stands, just with some
different details in how/when/where it fails.


===
Mark Millard
markmi at dsl-only.net