Starting to cross-compile non-base software for MIPS - what it's like

[Adrian Chadd]

On 15 June 2015 at 10:23, Warner Losh <imp at bsdimp.com> wrote:
>
>> On Jun 14, 2015, at 1:13 PM, Adrian Chadd <adrian at freebsd.org> wrote:
>>
>> Hi!
>>
>> I've been hacking at cross compiling things on FreeBSD for a while,
>> and I have made some progress. This email is less of a "how you can do
>> it" and more of a "these are the roadblocks that prevent it from
>> working out of the box."
>>
>> I'm using gcc-4.9.2 from ports - there's a cross compiler for
>> mips/mips64, and there's mips-binutils/mips64-binutils. I'm using
>> those to compile world and kernel. The kernel works; world requires
>> some patching to compile. I'll write a post when I have a complete
>> system booting/running from gcc-4.9.2.
>>
>> My test application is dropbear. Ie, I'm cross-compiling dropbear for
>> mips from freebsd/i386.
>>
>> * There are some issues in libgcc - I have a patch that I've emailed
>> out for review. I'll do some more testing and commit it next week when
>> I know it works.
>> * There are some environment variables that need to be set before I
>> compile dropbear. Here's what it looks like:
>>
>> CPU_ARGS="-march=mips32 -msoft-float -Wa,-msoft-float"
>> INCS="-I/home/adrian/work/freebsd/head-embedded-2/root/mips/usr/include"
>>
>> export CROSS_COMPILE=mips-portbld-freebsd11.0
>> export CC=${CROSS_COMPILE}-gcc
>> export CXX=${CROSS_COMPILE}-g++
>> #export LD=${CROSS_COMPILE}-ld
>> #export AR=${CROSS_COMPILE}-ar
>> #export RANLIB=${CROSS_COMPILE}-ranlib
>> #export STRIP=${CROSS_COMPILE}-strip
>> export LD=mips-freebsd-l
>> export AR=mips-freebsd-ar
>> export RANLIB=mips-freebsd-ranlib
>> export STRIP=mips-freebsd-strip
>> export SYSROOT=/home/adrian/work/freebsd/head-embedded-2/root/mips/
>> export CFLAGS="--sysroot=$SYSROOT ${CPU_ARGS} ${INCS} -O"
>> export CXXFLAGS="--sysroot=$SYSROOT ${CPU_ARGS} ${INCS} -O"
>> export CPPFLAGS="--sysroot=$SYSROOT ${CPU_ARGS} ${INCS} -O"
>> export LDFLAGS=--sysroot=$SYSROOT
>>
>> # now run:
>> # /configure --host=mips-portbld-freebsd11.0
>> # gmake
>>
>> ... then I hit a problem with how we populate links in /usr/lib and
>> such. TL;DR - the symlinks we use are not relative paths, they're
>> absolute paths - and this makes cross building using sysroot /
>> explicit library paths fail to work right. Sigh. In particular, I
>> needed to manually undo the libgcc symlink for it to compile.
>> compilation complained about other libraries, but the resulting
>> binaries did run.
>
> That’s one of about 6 reasons that what we produce isn’t a
> sysroot. We need to create a sysroot target that’s very close
> to the xdev targets with all the compiler bits stripped out. We’re
> close though. I have about 1/2 of the patch needed to remove
> xdev entirely and replace it with sysroot.

Sweet.

> You shouldn’t need the INCS args if you have a proper sysroot.
> And the CPU_ARGS is an indication we’re building our port
> incorrectly. This is the same feedback I gave on the kernel
> make changes you suggested.

It looks like a proper sysroot - I think it's just the dynamic sysroot
stuff being broken.

> I’ve had hacks like this for “simple” arts for years. It’s why I wrote
> xdev in the first place. I had it working in the FreeBSD 6 time frame
> for cross building some ports. But once you get past a certain
> class of very well behaved ports, it goes to hell in a hurry.


>> * I'll try to wrangle juli/bsdimp to review the kernel changes needed
>> to compile up the kernel with gcc-4.9.
>
> I’ve already reviewed that one. gcc isn’t being compiled correctly
> and that need to be fixed. This is the forth time I’ve given this
> feedback.

There are some other issues too. The softfloat/hardfloat stuff reared
its head - in order to avoid assembler warnings we have to manually
set it, but then the floating point register accesses in fp.S /
swtch.S generate errors. I've worked around that for now, but I need
to dig into why the warnings are generated in the first case.

>> * I'd like some help / comments on what we can do about the absolute
>> symlinks used for various things inside an installed root. Ideally
>> we'd have an option that would let us populate only relative links -
>> that'd instantly make this problem go away.
>
> All installworld-ish targets should use relative path symbolic links.
> The fact they are using absolute paths is a bug. In the FreeBSD 6
> time frame, I fought this battle at work and it wasn’t hard to fix. I
> no longer have access to those sources, and I honestly can’t
> recall how I fixed it (it may have been in a post-processing
> step).

I'd love to get this fixed and into the tree.

>> * I'll go hit up bapt and some netbsd friends for some help with
>> expected sysroot behaviour on gcc/binutils. It's possible there are
>> still bugs with the dynamic sysroot (ie, when you don't compile your
>> toolchain with an explicit sysroot to something other than '/') that
>> need addressing.
>
> bapt and I have been talking about this for a while. We’ve both been
> busy doing other things so haven’t focused on it other than in off
> moments.
>
>> Then I'll worry about starting to convert a few ports to be native
>> cross building.
>
> You should talk to bapt about the rather extensive work he’s done
> in this area, which is why I’ve looked at transitioning the xdev
> support to just sysroot support.

I have been too! fear not. That's how I got this far!


-a