11.0-CURRENT (r292413) on a rpi2b: arm-gnueabi-freebsd/bin/ar, _fseeko, and memset vs memory alignment (SCTRL bit[1]=1?): Explains the Bus error?
Warner Losh
imp at bsdimp.com
Fri Dec 25 19:53:54 UTC 2015
So what happens if we actually fix the underlying bug?
I see two ways of doing this. In findfp.c, we allocate an array of FILE * today like:
g = (struct glue *)malloc(sizeof(*g) + ALIGNBYTES + n * sizeof(FILE));
but that assumes that FILE just has normal pointer alignment requirements. However,
due to the mbstate having int64_t alignment requirements, this is wrong. Maybe we
need to do something like
g = (struct glue *)malloc(sizeof(*g) + max(sizeof(int64_t),ALIGNBYTES) + n * sizeof(FILE));
which wouldn’t change anything on LP64 systems, but would result in proper alignment
for ILP32 systems. We’d have to fix the loop that uses ALIGN afterwards to use
roundup. Instead, we’d need to round up to the neared 8-byte aligned offset (or technically,
the max of ALIGNBYTES and 8, but that’s always 8 on today’s systems. If we do this,
we can make sure that each file is 8-byte aligned or better. We may need to round up
sizeof(FILE) to a multiple of 8 as well. I believe that since it has the 8-byte alignment
for a member, its size must be a multiple of 8, but I’ve not chased that belief to ground.
If not, we may need another decorator (__aligned(8), I think, spelled with the ugly
max expression above). That way, the contract we’re making with the compiler will
always be true. ALIGN BYTES is 4 on Arm anyway, so that bit is clearly wrong.
This wouldn’t be an ABI change, since you can only get a valid FILE * from fopen (and
friends), plus stdin, stdout, and stderr. Those addresses aren’t hard coded into binaries,
so even if we have to tweak the last three and deal with some ‘fake’ FILE abuse in libc
(which I don’t think suffers from this issue, btw, given the alignment requirements that would
naturally follow from something on the stack), we’d still be ahead. At least for all CONFORMING
implementations[*]...
TL;DR: Why not make FILE * always 8-byte aligned? The compiler options are a band-aide.
Warner
[*] There’s at least on popular package that has a copy of the FILE structure in one of its
.h files and uses that to do unnatural optimization things, but even that’s cool, I think,
since it never allocates a new one.
> On Dec 25, 2015, at 7:24 AM, Mark Millard <markmi at dsl-only.net> wrote:
>
> [Good News Summary: Rebuilding buildworld/buildkernel for rpi2 11.0-CURRENT 292413 from amd64 based on adding -fmax-type-align=4 has so far removed the crashes during the toolchain activity: no more misaligned accesses in libc's _fseeko or elsewhere.]
>
> On 2015-Dec-25, at 12:31 AM, Mark Millard <markmi at dsl-only.net> wrote:
>
>> On 2015-Dec-24, at 10:39 PM, Mark Millard <markmi at dsl-only.net> wrote:
>>
>>> [I do not know if this partial crash analysis related to on-arm clang-associated activity is good enough and appropriate to submit or not.]
>>>
>>> The /usr/local/arm-gnueabi-freebsd/bin/ar on the rpi2b involved below came from pkg install activity instead of port building. Used as-is.
>>>
>>> When I just tried my first from-rpi2b builds (ports for a rpi2b), /usr/local/arm-gnueabi-freebsd/bin/ar crashed. I believe that the following suggests an alignment error for the type of instructions that memset for 128 bytes was translated to (sizeof(mbstate_t)) in the code used by /usr/local/arm-gnueabi-freebsd/bin/ar. (But I do not know how to check SCTLR bit[1] to be directly sure that alignment was being enforced.)
>>>
>>> The crash was a Bus error in /usr/local/arm-gnueabi-freebsd/bin/ar :
>>>
>>>> libtool: link: /usr/local/arm-gnueabi-freebsd/bin/ar cru .libs/libgnuintl.a bindtextdom.o dcgettext.o dgettext.o gettext.o finddomain.o hash-string.o loadmsgcat.o localealias.o textdomain.o l10nflist.o explodename.o dcigettext.o dcngettext.o dngettext.o ngettext.o pluralx.o plural-exp.o localcharset.o threadlib.o lock.o relocatable.o langprefs.o localename.o log.o printf.o setlocale.o version.o xsize.o osdep.o intl-compat.o
>>>> Bus error (core dumped)
>>>> *** [libgnuintl.la] Error code 138
>>>
>>> It failed in _fseeko doing a memset that turned into uses of "vst1.64 {d16-d17}, [r0]" instructions, for an address in register r0 that ended in 0xa4, so was not aligned to 8 byte boundaries. From what I read such "VSTn (multiple n-element structures)" that have .64 require 8 byte alignment. The evidence of the code and register value follow.
>>>
>>>> # gdb /usr/local/arm-gnueabi-freebsd/bin/ar /usr/obj/portswork/usr/ports/devel/gettext-tools/work/gettext-0.19.6/gettext-tools/intl/ar.core
>>>> . . .
>>>> #0 0x2033adcc in _fseeko (fp=0x20651dcc, offset=<value optimized out>, whence=<value optimized out>, ltest=<value optimized out>) at /usr/src/lib/libc/stdio/fseek.c:299
>>>> 299 memset(&fp->_mbstate, 0, sizeof(mbstate_t));
>>>> . . .
>>>> (gdb) x/24i 0x2033adb0
>>>> 0x2033adb0 <_fseeko+836>: vmov.i32 q8, #0 ; 0x00000000
>>>> 0x2033adb4 <_fseeko+840>: movw r1, #65503 ; 0xffdf
>>>> 0x2033adb8 <_fseeko+844>: stm r4, {r0, r7}
>>>> 0x2033adbc <_fseeko+848>: ldrh r0, [r4, #12]
>>>> 0x2033adc0 <_fseeko+852>: and r0, r0, r1
>>>> 0x2033adc4 <_fseeko+856>: strh r0, [r4, #12]
>>>> 0x2033adc8 <_fseeko+860>: add r0, r4, #216 ; 0xd8
>>>> 0x2033adcc <_fseeko+864>: vst1.64 {d16-d17}, [r0]
>>>> 0x2033add0 <_fseeko+868>: add r0, r4, #200 ; 0xc8
>>>> 0x2033add4 <_fseeko+872>: vst1.64 {d16-d17}, [r0]
>>>> 0x2033add8 <_fseeko+876>: add r0, r4, #184 ; 0xb8
>>>> 0x2033addc <_fseeko+880>: vst1.64 {d16-d17}, [r0]
>>>> 0x2033ade0 <_fseeko+884>: add r0, r4, #168 ; 0xa8
>>>> 0x2033ade4 <_fseeko+888>: vst1.64 {d16-d17}, [r0]
>>>> 0x2033ade8 <_fseeko+892>: add r0, r4, #152 ; 0x98
>>>> 0x2033adec <_fseeko+896>: vst1.64 {d16-d17}, [r0]
>>>> 0x2033adf0 <_fseeko+900>: add r0, r4, #136 ; 0x88
>>>> 0x2033adf4 <_fseeko+904>: vst1.64 {d16-d17}, [r0]
>>>> 0x2033adf8 <_fseeko+908>: add r0, r4, #120 ; 0x78
>>>> 0x2033adfc <_fseeko+912>: vst1.64 {d16-d17}, [r0]
>>>> 0x2033ae00 <_fseeko+916>: add r0, r4, #104 ; 0x68
>>>> 0x2033ae04 <_fseeko+920>: vst1.64 {d16-d17}, [r0]
>>>> 0x2033ae08 <_fseeko+924>: b 0x2033b070 <_fseeko+1540>
>>>> 0x2033ae0c <_fseeko+928>: cmp r5, #0 ; 0x0
>>>> (gdb) info all-registers
>>>> r0 0x20651ea4 543497892
>>>> r1 0xffdf 65503
>>>> r2 0x0 0
>>>> r3 0x0 0
>>>> r4 0x20651dcc 543497676
>>>> r5 0x0 0
>>>> r6 0x0 0
>>>> r7 0x0 0
>>>> r8 0x20359df4 540384756
>>>> r9 0x0 0
>>>> r10 0x0 0
>>>> r11 0xbfbfb948 -1077954232
>>>> r12 0x2037b208 540520968
>>>> sp 0xbfbfb898 -1077954408
>>>> lr 0x2035a004 540385284
>>>> pc 0x2033adcc 540257740
>>>> f0 0 (raw 0x000000000000000000000000)
>>>> f1 0 (raw 0x000000000000000000000000)
>>>> f2 0 (raw 0x000000000000000000000000)
>>>> f3 0 (raw 0x000000000000000000000000)
>>>> f4 0 (raw 0x000000000000000000000000)
>>>> f5 0 (raw 0x000000000000000000000000)
>>>> f6 0 (raw 0x000000000000000000000000)
>>>> f7 0 (raw 0x000000000000000000000000)
>>>> fps 0x0 0
>>>> cpsr 0x60000010 1610612752
>>>
>>> The syntax in use for vst1.64 instructions does not explicitly have the alignment notation. Presuming that the decoding is correct then from what I read the following applies:
>>>
>>>> Home > NEON and VFP Programming > NEON load and store element and structure instructions > Alignment restrictions in load and store, element and structure instructions
>>>>
>>>> . . . When the alignment is not specified in the instruction, the alignment restriction is controlled by the A bit (SCTLR bit[1]):
>>>> • if the A bit is 0, there are no alignment restrictions (except for strongly ordered or device memory, where accesses must be element aligned or the result is unpredictable)
>>>> • if the A bit is 1, accesses must be element aligned.
>>>> If an address is not correctly aligned, an alignment fault occurs.
>>>
>>> So if at the time the "A bit" (SCTLR bit[1]) is 1 then the Bus error would have the context to happen because of the mis-alignment.
>>>
>>> The following shows the make.conf context that explains how /usr/local/arm-gnueabi-freebsd/bin/ar came to be invoked:
>>>
>>>> # more /etc/make.conf
>>>> WRKDIRPREFIX=/usr/obj/portswork
>>>> WITH_DEBUG=
>>>> WITH_DEBUG_FILES=
>>>> MALLOC_PRODUCTION=
>>>> #
>>>> TO_TYPE=armv6
>>>> TOOLS_TO_TYPE=arm-gnueabi
>>>> CROSS_BINUTILS_PREFIX=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/
>>>> .if ${.MAKE.LEVEL} == 0
>>>> CC=/usr/bin/clang -target ${TO_TYPE}--freebsd11.0-gnueabi -march=armv7a
>>>> CXX=/usr/bin/clang++ -target ${TO_TYPE}--freebsd11.0-gnueabi -march=armv7a
>>>> CPP=/usr/bin/clang-cpp -target ${TO_TYPE}--freebsd11.0-gnueabi -march=armv7a
>>>> .export CC
>>>> .export CXX
>>>> .export CPP
>>>> AS=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/as
>>>> AR=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ar
>>>> LD=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ld
>>>> NM=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/nm
>>>> OBJCOPY=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/objcopy
>>>> OBJDUMP=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/objdump
>>>> RANLIB=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ranlib
>>>> SIZE=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/size
>>>> #NO-SUCH: STRINGS=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/strings
>>>> STRINGS=/usr/local/bin/${TOOLS_TO_TYPE}-freebsd-strings
>>>> .export AS
>>>> .export AR
>>>> .export LD
>>>> .export NM
>>>> .export OBJCOPY
>>>> .export OBJDUMP
>>>> .export RANLIB
>>>> .export SIZE
>>>> .export STRINGS
>>>> .endif
>>>
>>>
>>> Other context:
>>>
>>>> # freebsd-version -ku; uname -aKU
>>>> 11.0-CURRENT
>>>> 11.0-CURRENT
>>>> FreeBSD rpi2 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r292413M: Tue Dec 22 22:02:21 PST 2015 root at FreeBSDx64:/usr/obj/clang/arm.armv6/usr/src/sys/RPI2-NODBG arm 1100091 1100091
>>>
>>>
>>>
>>> I will note that world and kernel are my own build of -r292413 (earlier experiment) --a build made from an amd64 host context and put in place via DESTDIR=. My expectation would be that the amd64 context would not be likely to have similar alignment restrictions involved in its ar activity (or other activity). That would explain how I got this far using such a clang 3.7 related toolchain for targeting an rpi2 before finding such a problem.
>>
>>
>> I realized re-reading the all above that it seems to suggest that the _fseeko code involved is from /usr/local/arm-gnueabi-freebsd/bin/ar but that was not my intent.
>>
>> libc.so.7 is from my buildworld, including the fseeko implementation:
>>
>> Reading symbols from /lib/libc.so.7...Reading symbols from /usr/lib/debug//lib/libc.so.7.debug...done.
>> done.
>> Loaded symbols for /lib/libc.so.7
>>
>>
>> head/sys/sys/_types.h has:
>>
>> /*
>> * mbstate_t is an opaque object to keep conversion state during multibyte
>> * stream conversions.
>> */
>> typedef union {
>> char __mbstate8[128];
>> __int64_t _mbstateL; /* for alignment */
>> } __mbstate_t;
>>
>> suggesting an implicit alignment of the union to whatever the implementation defines for __int64_t --which need not be 8 byte alignment (in the abstract, general case). But 8 byte alignment is a possibility as well (in the abstract).
>>
>> But printing *fp in gdb for the fp argument to _fseeko reports the same not-8-byte aligned address for __mbstate8 that was in r0:
>>
>>> (gdb) bt
>>> #0 0x2033adcc in _fseeko (fp=0x20651dcc, offset=<value optimized out>, whence=<value optimized out>, ltest=<value optimized out>) at /usr/src/lib/libc/stdio/fseek.c:299
>>> #1 0x2033b108 in fseeko (fp=0x20651dcc, offset=18571438587904, whence=0) at /usr/src/lib/libc/stdio/fseek.c:82
>>> #2 0x00016138 in ?? ()
>>> (gdb) print fp
>>> $2 = (FILE *) 0x20651dcc
>>> (gdb) print *fp
>>> $3 = {_p = 0x2069a240 "", _r = 0, _w = 0, _flags = 5264, _file = 36, _bf = {_base = 0x2069a240 "", _size = 32768}, _lbfsize = 0, _cookie = 0x20651dcc, _close = 0x20359dfc <__sclose>,
>>> _read = 0x20359de4 <__sread>, _seek = 0x20359df4 <__sseek>, _write = 0x20359dec <__swrite>, _ub = {_base = 0x0, _size = 0}, _up = 0x0, _ur = 0, _ubuf = 0x20651e0c "", _nbuf = 0x20651e0f "", _lb = {
>>> _base = 0x0, _size = 0}, _blksize = 32768, _offset = 0, _fl_mutex = 0x0, _fl_owner = 0x0, _fl_count = 0, _orientation = 0, _mbstate = {__mbstate8 = 0x20651e34 "", _mbstateL = 0}, _flags2 = 0}
>>
>> The overall FILE struct containing the _mbstate field is also not 8-byte aligned. But the offset from the start of the FILE struct to __mbstate8 is a multiple of 8 bytes.
>>
>> It is my interpretation that there is nothing here to justify the memset implementation combination:
>>
>> SCTLR bit[1]==1
>>
>> mixed with
>>
>> vst1.64 instructions
>>
>> I.e.: one or both needs to change unless some way for forcing 8-byte alignment is introduced.
>>
>> I have not managed to track down anything that would indicate FreeBSD's intent for SCTLR bit[1]. I do not even know if it is required by the design to be constant (once initialized).
>
>
> I have (so far) removed the build tool crashes based on adding -fmax-type-align=4 to avoid the misaligned accesses. Details follow.
>
> src.conf on amd64 for the rpi2 targeting buildworld/buildkernel now looks like:
>
>> # more ~/src.configs/src.conf.rpi2-clang.amd64-host
>> TO_TYPE=armv6
>> TOOLS_TO_TYPE=arm-gnueabi
>> FROM_TYPE=amd64
>> TOOLS_FROM_TYPE=x86_64
>> VERSION_CONTEXT=11.0
>> #
>> KERNCONF=RPI2-NODBG
>> TARGET=arm
>> .if ${.MAKE.LEVEL} == 0
>> TARGET_ARCH=${TO_TYPE}
>> .export TARGET_ARCH
>> .endif
>> #
>> WITHOUT_CROSS_COMPILER=
>> #
>> # For WITH_BOOT= . . .
>> # arm-gnueabi-freebsd/bin/ld reports bootinfo.o: relocation R_ARM_MOVW_ABS_NC against `a local symbol' can not be used when making a shared object; recompile with -fPIC
>> WITHOUT_BOOT=
>> #
>> WITH_FAST_DEPEND=
>> WITH_LIBCPLUSPLUS=
>> WITH_CLANG=
>> WITH_CLANG_IS_CC=
>> WITH_CLANG_FULL=
>> WITH_LLDB=
>> WITH_CLANG_EXTRAS=
>> #
>> WITHOUT_LIB32=
>> WITHOUT_GCC=
>> WITHOUT_GNUCXX=
>> #
>> NO_WERROR=
>> MALLOC_PRODUCTION=
>> #CFLAGS+= -DELF_VERBOSE
>> #
>> WITH_DEBUG=
>> WITH_DEBUG_FILES=
>> #
>> # TOOLS_TO_TYPE based on ${TO_TYPE}-xtoolchain-gcc related bintutils...
>> #
>> #CROSS_TOOLCHAIN=${TO_TYPE}-gcc
>> X_COMPILER_TYPE=clang
>> CROSS_BINUTILS_PREFIX=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/
>> .if ${.MAKE.LEVEL} == 0
>> XCC=/usr/bin/clang -target ${TO_TYPE}--freebsd11.0-gnueabi -march=armv7a -fmax-type-align=4
>> XCXX=/usr/bin/clang++ -target ${TO_TYPE}--freebsd11.0-gnueabi -march=armv7a -fmax-type-align=4
>> XCPP=/usr/bin/clang-cpp -target ${TO_TYPE}--freebsd11.0-gnueabi -march=armv7a -fmax-type-align=4
>> .export XCC
>> .export XCXX
>> .export XCPP
>> XAS=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/as
>> XAR=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ar
>> XLD=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ld
>> XNM=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/nm
>> XOBJCOPY=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/objcopy
>> XOBJDUMP=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/objdump
>> XRANLIB=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ranlib
>> XSIZE=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/size
>> #NO-SUCH: XSTRINGS=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/strings
>> XSTRINGS=/usr/local/bin/${TOOLS_TO_TYPE}-freebsd-strings
>> .export XAS
>> .export XAR
>> .export XLD
>> .export XNM
>> .export XOBJCOPY
>> .export XOBJDUMP
>> .export XRANLIB
>> .export XSIZE
>> .export XSTRINGS
>> .endif
>> #
>> # Host compiler stuff:
>> .if ${.MAKE.LEVEL} == 0
>> CC=/usr/bin/clang -B/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin
>> CXX=/usr/bin/clang++ -B/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin
>> CPP=/usr/bin/clang-cpp -B/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin
>> .export CC
>> .export CXX
>> .export CPP
>> AS=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/as
>> AR=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/ar
>> LD=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/ld
>> NM=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/nm
>> OBJCOPY=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/objcopy
>> OBJDUMP=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/objdump
>> RANLIB=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/ranlib
>> SIZE=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/size
>> #NO-SUCH: STRINGS=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/strings
>> STRINGS=/usr/local/bin/${TOOLS_FROM_TYPE}-freebsd-strings
>> .export AS
>> .export AR
>> .export LD
>> .export NM
>> .export OBJCOPY
>> .export OBJDUMP
>> .export RANLIB
>> .export SIZE
>> .export STRINGS
>> .endif
>
> make.conf for during the on-rpi2 port builds now looks like:
>
>> $ more /etc/make.conf
>> WRKDIRPREFIX=/usr/obj/portswork
>> WITH_DEBUG=
>> WITH_DEBUG_FILES=
>> MALLOC_PRODUCTION=
>> #
>> TO_TYPE=armv6
>> TOOLS_TO_TYPE=arm-gnueabi
>> CROSS_BINUTILS_PREFIX=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/
>> .if ${.MAKE.LEVEL} == 0
>> CC=/usr/bin/clang -target ${TO_TYPE}--freebsd11.0-gnueabi -march=armv7a -fmax-type-align=4
>> CXX=/usr/bin/clang++ -target ${TO_TYPE}--freebsd11.0-gnueabi -march=armv7a -fmax-type-align=4
>> CPP=/usr/bin/clang-cpp -target ${TO_TYPE}--freebsd11.0-gnueabi -march=armv7a -fmax-type-align=4
>> .export CC
>> .export CXX
>> .export CPP
>> AS=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/as
>> AR=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ar
>> LD=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ld
>> NM=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/nm
>> OBJCOPY=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/objcopy
>> OBJDUMP=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/objdump
>> RANLIB=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ranlib
>> SIZE=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/size
>> #NO-SUCH: STRINGS=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/strings
>> STRINGS=/usr/local/bin/${TOOLS_TO_TYPE}-freebsd-strings
>> .export AS
>> .export AR
>> .export LD
>> .export NM
>> .export OBJCOPY
>> .export OBJDUMP
>> .export RANLIB
>> .export SIZE
>> .export STRINGS
>> .endif
>
>
>
> ===
> Mark Millard
> markmi at dsl-only.net
>
>
>
> _______________________________________________
> freebsd-toolchain at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
> To unsubscribe, send any mail to "freebsd-toolchain-unsubscribe at freebsd.org"
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.freebsd.org/pipermail/freebsd-arm/attachments/20151225/ee224be9/attachment.sig>
More information about the freebsd-arm
mailing list