Cross-compile worked, cross-install not so much ...

From: Patrick M. Hausen <pmh_at_hausen.com>
Date: Mon, 25 Apr 2022 19:07:47 UTC
Hi all,

getting into FreeBSD ARM64 I tried to compile a current system on a fast AMD64 VM and
now I am somewhat stuck - no help using search engines and the like.

1st step:

	# checkout main aka 14-CURRENT
	cd /usr/src
	make -j 8 TARGET=arm64 TARGET_ARCH=aarch64 buildworld buildkernel

That worked like a charm and I seem to have a well populated /usr/obj/usr/src/arm64.aarch64.

2nd step:

I then mounted /usr/src and /usr/obj on my Raspberry Pi via NFS. The binaries
in /usr/obj are the correct architecture and run perfectly well:

	root@pi8:~ # file /usr/obj/usr/src/arm64.aarch64/bin/sh/sh
	/usr/obj/usr/src/arm64.aarch64/bin/sh/sh: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 14.0 (1400057), FreeBSD-style, not stripped
	root@pi8:~ # /usr/obj/usr/src/arm64.aarch64/bin/sh/sh
	root@pi8:~ # ^D

3rd step:

	root@pi8:/usr/src # make TARGET=arm64 TARGET_ARCH=aarch64 installkernel
	--------------------------------------------------------------
	>>> Install check kernel
	--------------------------------------------------------------
	--------------------------------------------------------------
	>>> Installing kernel GENERIC on Mon Apr 25 21:03:58 CEST 2022
	--------------------------------------------------------------
	cd /usr/obj/usr/src/arm64.aarch64/sys/GENERIC;  MACHINE_ARCH=aarch64  MACHINE=arm64  CPUTYPE= CC="cc -target aarch64-unknown-freebsd14.0 --sysroot=/usr/obj/usr/src/arm64.aarch64/tmp -B/usr/obj/usr/src/arm64.aarch64/tmp/usr/bin" CXX="c++  -target aarch64-unknown-freebsd14.0 --sysroot=/usr/obj/usr/src/arm64.aarch64/tmp -B/usr/obj/usr/src/arm64.aarch64/tmp/usr/bin"  CPP="cpp -target aarch64-unknown-freebsd14.0 --sysroot=/usr/obj/usr/src/arm64.aarch64/tmp -B/usr/obj/usr/src/arm64.aarch64/tmp/usr/bin"  AS="as" AR="ar" ELFCTL="elfctl" LD="ld"  LLVM_LINK="" NM=nm OBJCOPY="objcopy"  RANLIB=ranlib STRINGS=  SIZE="size" STRIPBIN="strip" PATH=/usr/obj/usr/src/arm64.aarch64/tmp/bin:/usr/obj/usr/src/arm64.aarch64/tmp/usr/sbin:/usr/obj/usr/src/arm64.aarch64/tmp/usr/bin:/usr/obj/usr/src/arm64.aarch64/tmp/legacy/usr/sbin:/usr/obj/usr/src/arm64.aarch64/tmp/legacy/usr/bin:/usr/obj/usr/src/arm64.aarch64/tmp/legacy/bin:/usr/obj/usr/src/arm64.aarch64/tmp/legacy/usr/libexec::/sbin:/bin:/usr/sbin:/usr/bin  make  KERNEL=kernel install
	/bin/sh: make: Exec format error
	*** Error code 126

	Stop.
	make[1]: stopped in /usr/src
	*** Error code 1

	Stop.
	make: stopped in /usr/src

So via that complicated PATH setting it seems to run: /usr/obj/usr/src/arm64.aarch64/tmp/legacy/usr/bin/make

Which surprisingly is an amd64 binary:

	root@pi8:/usr/src # file /usr/obj/usr/src/arm64.aarch64/tmp/legacy/usr/bin/make
	/usr/obj/usr/src/arm64.aarch64/tmp/legacy/usr/bin/make: ELF 64-bit LSB pie executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 14.0 (1400057), FreeBSD-style, stripped


So what did I do wrong?

Thanks for all help,
Patrick