11.0-CURRENT (r292413) on a rpi2b: arm-gnueabi-freebsd/bin/ar, _fseeko, and memset vs memory alignment (SCTRL bit[1]=1?): Explains the Bus error?
Mark Millard
markmi at dsl-only.net
Fri Dec 25 14:25:01 UTC 2015
[Good News Summary: Rebuilding buildworld/buildkernel for rpi2 11.0-CURRENT 292413 from amd64 based on adding -fmax-type-align=4 has so far removed the crashes during the toolchain activity: no more misaligned accesses in libc's _fseeko or elsewhere.]
On 2015-Dec-25, at 12:31 AM, Mark Millard <markmi at dsl-only.net> wrote:
> On 2015-Dec-24, at 10:39 PM, Mark Millard <markmi at dsl-only.net> wrote:
>
>> [I do not know if this partial crash analysis related to on-arm clang-associated activity is good enough and appropriate to submit or not.]
>>
>> The /usr/local/arm-gnueabi-freebsd/bin/ar on the rpi2b involved below came from pkg install activity instead of port building. Used as-is.
>>
>> When I just tried my first from-rpi2b builds (ports for a rpi2b), /usr/local/arm-gnueabi-freebsd/bin/ar crashed. I believe that the following suggests an alignment error for the type of instructions that memset for 128 bytes was translated to (sizeof(mbstate_t)) in the code used by /usr/local/arm-gnueabi-freebsd/bin/ar. (But I do not know how to check SCTLR bit[1] to be directly sure that alignment was being enforced.)
>>
>> The crash was a Bus error in /usr/local/arm-gnueabi-freebsd/bin/ar :
>>
>>> libtool: link: /usr/local/arm-gnueabi-freebsd/bin/ar cru .libs/libgnuintl.a bindtextdom.o dcgettext.o dgettext.o gettext.o finddomain.o hash-string.o loadmsgcat.o localealias.o textdomain.o l10nflist.o explodename.o dcigettext.o dcngettext.o dngettext.o ngettext.o pluralx.o plural-exp.o localcharset.o threadlib.o lock.o relocatable.o langprefs.o localename.o log.o printf.o setlocale.o version.o xsize.o osdep.o intl-compat.o
>>> Bus error (core dumped)
>>> *** [libgnuintl.la] Error code 138
>>
>> It failed in _fseeko doing a memset that turned into uses of "vst1.64 {d16-d17}, [r0]" instructions, for an address in register r0 that ended in 0xa4, so was not aligned to 8 byte boundaries. From what I read such "VSTn (multiple n-element structures)" that have .64 require 8 byte alignment. The evidence of the code and register value follow.
>>
>>> # gdb /usr/local/arm-gnueabi-freebsd/bin/ar /usr/obj/portswork/usr/ports/devel/gettext-tools/work/gettext-0.19.6/gettext-tools/intl/ar.core
>>> . . .
>>> #0 0x2033adcc in _fseeko (fp=0x20651dcc, offset=<value optimized out>, whence=<value optimized out>, ltest=<value optimized out>) at /usr/src/lib/libc/stdio/fseek.c:299
>>> 299 memset(&fp->_mbstate, 0, sizeof(mbstate_t));
>>> . . .
>>> (gdb) x/24i 0x2033adb0
>>> 0x2033adb0 <_fseeko+836>: vmov.i32 q8, #0 ; 0x00000000
>>> 0x2033adb4 <_fseeko+840>: movw r1, #65503 ; 0xffdf
>>> 0x2033adb8 <_fseeko+844>: stm r4, {r0, r7}
>>> 0x2033adbc <_fseeko+848>: ldrh r0, [r4, #12]
>>> 0x2033adc0 <_fseeko+852>: and r0, r0, r1
>>> 0x2033adc4 <_fseeko+856>: strh r0, [r4, #12]
>>> 0x2033adc8 <_fseeko+860>: add r0, r4, #216 ; 0xd8
>>> 0x2033adcc <_fseeko+864>: vst1.64 {d16-d17}, [r0]
>>> 0x2033add0 <_fseeko+868>: add r0, r4, #200 ; 0xc8
>>> 0x2033add4 <_fseeko+872>: vst1.64 {d16-d17}, [r0]
>>> 0x2033add8 <_fseeko+876>: add r0, r4, #184 ; 0xb8
>>> 0x2033addc <_fseeko+880>: vst1.64 {d16-d17}, [r0]
>>> 0x2033ade0 <_fseeko+884>: add r0, r4, #168 ; 0xa8
>>> 0x2033ade4 <_fseeko+888>: vst1.64 {d16-d17}, [r0]
>>> 0x2033ade8 <_fseeko+892>: add r0, r4, #152 ; 0x98
>>> 0x2033adec <_fseeko+896>: vst1.64 {d16-d17}, [r0]
>>> 0x2033adf0 <_fseeko+900>: add r0, r4, #136 ; 0x88
>>> 0x2033adf4 <_fseeko+904>: vst1.64 {d16-d17}, [r0]
>>> 0x2033adf8 <_fseeko+908>: add r0, r4, #120 ; 0x78
>>> 0x2033adfc <_fseeko+912>: vst1.64 {d16-d17}, [r0]
>>> 0x2033ae00 <_fseeko+916>: add r0, r4, #104 ; 0x68
>>> 0x2033ae04 <_fseeko+920>: vst1.64 {d16-d17}, [r0]
>>> 0x2033ae08 <_fseeko+924>: b 0x2033b070 <_fseeko+1540>
>>> 0x2033ae0c <_fseeko+928>: cmp r5, #0 ; 0x0
>>> (gdb) info all-registers
>>> r0 0x20651ea4 543497892
>>> r1 0xffdf 65503
>>> r2 0x0 0
>>> r3 0x0 0
>>> r4 0x20651dcc 543497676
>>> r5 0x0 0
>>> r6 0x0 0
>>> r7 0x0 0
>>> r8 0x20359df4 540384756
>>> r9 0x0 0
>>> r10 0x0 0
>>> r11 0xbfbfb948 -1077954232
>>> r12 0x2037b208 540520968
>>> sp 0xbfbfb898 -1077954408
>>> lr 0x2035a004 540385284
>>> pc 0x2033adcc 540257740
>>> f0 0 (raw 0x000000000000000000000000)
>>> f1 0 (raw 0x000000000000000000000000)
>>> f2 0 (raw 0x000000000000000000000000)
>>> f3 0 (raw 0x000000000000000000000000)
>>> f4 0 (raw 0x000000000000000000000000)
>>> f5 0 (raw 0x000000000000000000000000)
>>> f6 0 (raw 0x000000000000000000000000)
>>> f7 0 (raw 0x000000000000000000000000)
>>> fps 0x0 0
>>> cpsr 0x60000010 1610612752
>>
>> The syntax in use for vst1.64 instructions does not explicitly have the alignment notation. Presuming that the decoding is correct then from what I read the following applies:
>>
>>> Home > NEON and VFP Programming > NEON load and store element and structure instructions > Alignment restrictions in load and store, element and structure instructions
>>>
>>> . . . When the alignment is not specified in the instruction, the alignment restriction is controlled by the A bit (SCTLR bit[1]):
>>> • if the A bit is 0, there are no alignment restrictions (except for strongly ordered or device memory, where accesses must be element aligned or the result is unpredictable)
>>> • if the A bit is 1, accesses must be element aligned.
>>> If an address is not correctly aligned, an alignment fault occurs.
>>
>> So if at the time the "A bit" (SCTLR bit[1]) is 1 then the Bus error would have the context to happen because of the mis-alignment.
>>
>> The following shows the make.conf context that explains how /usr/local/arm-gnueabi-freebsd/bin/ar came to be invoked:
>>
>>> # more /etc/make.conf
>>> WRKDIRPREFIX=/usr/obj/portswork
>>> WITH_DEBUG=
>>> WITH_DEBUG_FILES=
>>> MALLOC_PRODUCTION=
>>> #
>>> TO_TYPE=armv6
>>> TOOLS_TO_TYPE=arm-gnueabi
>>> CROSS_BINUTILS_PREFIX=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/
>>> .if ${.MAKE.LEVEL} == 0
>>> CC=/usr/bin/clang -target ${TO_TYPE}--freebsd11.0-gnueabi -march=armv7a
>>> CXX=/usr/bin/clang++ -target ${TO_TYPE}--freebsd11.0-gnueabi -march=armv7a
>>> CPP=/usr/bin/clang-cpp -target ${TO_TYPE}--freebsd11.0-gnueabi -march=armv7a
>>> .export CC
>>> .export CXX
>>> .export CPP
>>> AS=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/as
>>> AR=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ar
>>> LD=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ld
>>> NM=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/nm
>>> OBJCOPY=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/objcopy
>>> OBJDUMP=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/objdump
>>> RANLIB=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ranlib
>>> SIZE=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/size
>>> #NO-SUCH: STRINGS=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/strings
>>> STRINGS=/usr/local/bin/${TOOLS_TO_TYPE}-freebsd-strings
>>> .export AS
>>> .export AR
>>> .export LD
>>> .export NM
>>> .export OBJCOPY
>>> .export OBJDUMP
>>> .export RANLIB
>>> .export SIZE
>>> .export STRINGS
>>> .endif
>>
>>
>> Other context:
>>
>>> # freebsd-version -ku; uname -aKU
>>> 11.0-CURRENT
>>> 11.0-CURRENT
>>> FreeBSD rpi2 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r292413M: Tue Dec 22 22:02:21 PST 2015 root at FreeBSDx64:/usr/obj/clang/arm.armv6/usr/src/sys/RPI2-NODBG arm 1100091 1100091
>>
>>
>>
>> I will note that world and kernel are my own build of -r292413 (earlier experiment) --a build made from an amd64 host context and put in place via DESTDIR=. My expectation would be that the amd64 context would not be likely to have similar alignment restrictions involved in its ar activity (or other activity). That would explain how I got this far using such a clang 3.7 related toolchain for targeting an rpi2 before finding such a problem.
>
>
> I realized re-reading the all above that it seems to suggest that the _fseeko code involved is from /usr/local/arm-gnueabi-freebsd/bin/ar but that was not my intent.
>
> libc.so.7 is from my buildworld, including the fseeko implementation:
>
> Reading symbols from /lib/libc.so.7...Reading symbols from /usr/lib/debug//lib/libc.so.7.debug...done.
> done.
> Loaded symbols for /lib/libc.so.7
>
>
> head/sys/sys/_types.h has:
>
> /*
> * mbstate_t is an opaque object to keep conversion state during multibyte
> * stream conversions.
> */
> typedef union {
> char __mbstate8[128];
> __int64_t _mbstateL; /* for alignment */
> } __mbstate_t;
>
> suggesting an implicit alignment of the union to whatever the implementation defines for __int64_t --which need not be 8 byte alignment (in the abstract, general case). But 8 byte alignment is a possibility as well (in the abstract).
>
> But printing *fp in gdb for the fp argument to _fseeko reports the same not-8-byte aligned address for __mbstate8 that was in r0:
>
>> (gdb) bt
>> #0 0x2033adcc in _fseeko (fp=0x20651dcc, offset=<value optimized out>, whence=<value optimized out>, ltest=<value optimized out>) at /usr/src/lib/libc/stdio/fseek.c:299
>> #1 0x2033b108 in fseeko (fp=0x20651dcc, offset=18571438587904, whence=0) at /usr/src/lib/libc/stdio/fseek.c:82
>> #2 0x00016138 in ?? ()
>> (gdb) print fp
>> $2 = (FILE *) 0x20651dcc
>> (gdb) print *fp
>> $3 = {_p = 0x2069a240 "", _r = 0, _w = 0, _flags = 5264, _file = 36, _bf = {_base = 0x2069a240 "", _size = 32768}, _lbfsize = 0, _cookie = 0x20651dcc, _close = 0x20359dfc <__sclose>,
>> _read = 0x20359de4 <__sread>, _seek = 0x20359df4 <__sseek>, _write = 0x20359dec <__swrite>, _ub = {_base = 0x0, _size = 0}, _up = 0x0, _ur = 0, _ubuf = 0x20651e0c "", _nbuf = 0x20651e0f "", _lb = {
>> _base = 0x0, _size = 0}, _blksize = 32768, _offset = 0, _fl_mutex = 0x0, _fl_owner = 0x0, _fl_count = 0, _orientation = 0, _mbstate = {__mbstate8 = 0x20651e34 "", _mbstateL = 0}, _flags2 = 0}
>
> The overall FILE struct containing the _mbstate field is also not 8-byte aligned. But the offset from the start of the FILE struct to __mbstate8 is a multiple of 8 bytes.
>
> It is my interpretation that there is nothing here to justify the memset implementation combination:
>
> SCTLR bit[1]==1
>
> mixed with
>
> vst1.64 instructions
>
> I.e.: one or both needs to change unless some way for forcing 8-byte alignment is introduced.
>
> I have not managed to track down anything that would indicate FreeBSD's intent for SCTLR bit[1]. I do not even know if it is required by the design to be constant (once initialized).
I have (so far) removed the build tool crashes based on adding -fmax-type-align=4 to avoid the misaligned accesses. Details follow.
src.conf on amd64 for the rpi2 targeting buildworld/buildkernel now looks like:
> # more ~/src.configs/src.conf.rpi2-clang.amd64-host
> TO_TYPE=armv6
> TOOLS_TO_TYPE=arm-gnueabi
> FROM_TYPE=amd64
> TOOLS_FROM_TYPE=x86_64
> VERSION_CONTEXT=11.0
> #
> KERNCONF=RPI2-NODBG
> TARGET=arm
> .if ${.MAKE.LEVEL} == 0
> TARGET_ARCH=${TO_TYPE}
> .export TARGET_ARCH
> .endif
> #
> WITHOUT_CROSS_COMPILER=
> #
> # For WITH_BOOT= . . .
> # arm-gnueabi-freebsd/bin/ld reports bootinfo.o: relocation R_ARM_MOVW_ABS_NC against `a local symbol' can not be used when making a shared object; recompile with -fPIC
> WITHOUT_BOOT=
> #
> WITH_FAST_DEPEND=
> WITH_LIBCPLUSPLUS=
> WITH_CLANG=
> WITH_CLANG_IS_CC=
> WITH_CLANG_FULL=
> WITH_LLDB=
> WITH_CLANG_EXTRAS=
> #
> WITHOUT_LIB32=
> WITHOUT_GCC=
> WITHOUT_GNUCXX=
> #
> NO_WERROR=
> MALLOC_PRODUCTION=
> #CFLAGS+= -DELF_VERBOSE
> #
> WITH_DEBUG=
> WITH_DEBUG_FILES=
> #
> # TOOLS_TO_TYPE based on ${TO_TYPE}-xtoolchain-gcc related bintutils...
> #
> #CROSS_TOOLCHAIN=${TO_TYPE}-gcc
> X_COMPILER_TYPE=clang
> CROSS_BINUTILS_PREFIX=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/
> .if ${.MAKE.LEVEL} == 0
> XCC=/usr/bin/clang -target ${TO_TYPE}--freebsd11.0-gnueabi -march=armv7a -fmax-type-align=4
> XCXX=/usr/bin/clang++ -target ${TO_TYPE}--freebsd11.0-gnueabi -march=armv7a -fmax-type-align=4
> XCPP=/usr/bin/clang-cpp -target ${TO_TYPE}--freebsd11.0-gnueabi -march=armv7a -fmax-type-align=4
> .export XCC
> .export XCXX
> .export XCPP
> XAS=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/as
> XAR=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ar
> XLD=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ld
> XNM=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/nm
> XOBJCOPY=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/objcopy
> XOBJDUMP=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/objdump
> XRANLIB=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ranlib
> XSIZE=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/size
> #NO-SUCH: XSTRINGS=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/strings
> XSTRINGS=/usr/local/bin/${TOOLS_TO_TYPE}-freebsd-strings
> .export XAS
> .export XAR
> .export XLD
> .export XNM
> .export XOBJCOPY
> .export XOBJDUMP
> .export XRANLIB
> .export XSIZE
> .export XSTRINGS
> .endif
> #
> # Host compiler stuff:
> .if ${.MAKE.LEVEL} == 0
> CC=/usr/bin/clang -B/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin
> CXX=/usr/bin/clang++ -B/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin
> CPP=/usr/bin/clang-cpp -B/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin
> .export CC
> .export CXX
> .export CPP
> AS=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/as
> AR=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/ar
> LD=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/ld
> NM=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/nm
> OBJCOPY=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/objcopy
> OBJDUMP=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/objdump
> RANLIB=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/ranlib
> SIZE=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/size
> #NO-SUCH: STRINGS=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/strings
> STRINGS=/usr/local/bin/${TOOLS_FROM_TYPE}-freebsd-strings
> .export AS
> .export AR
> .export LD
> .export NM
> .export OBJCOPY
> .export OBJDUMP
> .export RANLIB
> .export SIZE
> .export STRINGS
> .endif
make.conf for during the on-rpi2 port builds now looks like:
> $ more /etc/make.conf
> WRKDIRPREFIX=/usr/obj/portswork
> WITH_DEBUG=
> WITH_DEBUG_FILES=
> MALLOC_PRODUCTION=
> #
> TO_TYPE=armv6
> TOOLS_TO_TYPE=arm-gnueabi
> CROSS_BINUTILS_PREFIX=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/
> .if ${.MAKE.LEVEL} == 0
> CC=/usr/bin/clang -target ${TO_TYPE}--freebsd11.0-gnueabi -march=armv7a -fmax-type-align=4
> CXX=/usr/bin/clang++ -target ${TO_TYPE}--freebsd11.0-gnueabi -march=armv7a -fmax-type-align=4
> CPP=/usr/bin/clang-cpp -target ${TO_TYPE}--freebsd11.0-gnueabi -march=armv7a -fmax-type-align=4
> .export CC
> .export CXX
> .export CPP
> AS=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/as
> AR=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ar
> LD=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ld
> NM=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/nm
> OBJCOPY=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/objcopy
> OBJDUMP=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/objdump
> RANLIB=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ranlib
> SIZE=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/size
> #NO-SUCH: STRINGS=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/strings
> STRINGS=/usr/local/bin/${TOOLS_TO_TYPE}-freebsd-strings
> .export AS
> .export AR
> .export LD
> .export NM
> .export OBJCOPY
> .export OBJDUMP
> .export RANLIB
> .export SIZE
> .export STRINGS
> .endif
===
Mark Millard
markmi at dsl-only.net
More information about the freebsd-arm
mailing list