I've shown the following 3 items via kyua testing with the latest snapshot of armv7 14 on a OrangePi+2Ed (Cortex-A7), a panic included
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Sun, 06 Aug 2023 01:56:59 UTC
I've set up armv7 boot media based on:
http://ftp3.freebsd.org/pub/FreeBSD/snapshots/ISO-IMAGES/14.0/FreeBSD-14.0-CURRENT-arm-armv7-GENERICSD-20230803-8a5c836b51ce-264491.img.xz
for the OrangePi+2Ed (it also handles the RPi2B v1.1).
No builds by me. The ports installed are from the FreeBSD servers and
are for kyua activity's use, other than gdb.
The presence of panic and large count of hours waiting for
timeouts explain why this report only covers specific issues
and no a full kyua run at this point.
[I'll note that this activity has been driven by attempted testing
of lib32, where I then end up with the question "is this unique to
lib32?" and so end up looking at armv7 chroot on aarch64. More
problems there --and so the question "is this unique to armv7 use
on aarch64". This leads to looking at Cortex-A7 armv7 (a fully
native armv7) context for comparison. I've had to wander well
outside my original intended testing contexts and have ended up
blocked in each type of context. Given the original goal, I'm
sticking to main, not stable/* nor releng/*.* .]
EXAMPLE issue: *.py based kyua testing problem for armv7 main:
The kyua *.py based tests on armv7 main get long timeouts
when python executes as armv7 code, even for very simple
tests:
# /usr/bin/kyua test -k /usr/tests/Kyuafile examples/test_examples.py:TestExampleSimple::test_with_cleanup
examples/test_examples.py:TestExampleSimple::test_with_cleanup -> broken: Test case body timed out [300.062s]
Results file id is usr_tests.20230806-003823-404601
Results saved to /root/.kyua/store/results.usr_tests.20230806-003823-404601.db
0/1 passed (1 failed)
This happens even on the cortex-A7 system, no lib32 involved,
no chroot involved.
NOTE:
There are a lot of these tests and some have timeouts set at
3600s, others at 1800s and 1200s. Lots at 300s. If a full
kyua run completed, it would take a very long time to do so.
Kyua classifies all these long-timeout tests as broken. So
lots of testing is not happening, despte the time taken.
There are 10 or so:
-> skipped: comment me to run the test
and one each of each of:
-> skipped: Current architecture 'armv7' not supported
__test_cases_list__ -> broken: Test program did not exit cleanly
__test_cases_list__ -> broken: Test case list wrote to stderr
I do not know if this promblem type might be somehow tied to the
openssl 3 compatibility issue(s) that the ports python's currently
have for main. If it is, the error handling seems to not be
directly reporting anything that makes it obvious.
EXAMPLE armv7 related 'Alignment Fault' on read panic:
The kyua sys/netinet6/exthdr:exthdr panic has a different backtrace than
I've had from my builds, udp_input this time:
# /usr/bin/kyua test -k /usr/tests/Kyuafile sys/netinet6/exthdr:exthdr
sys/netinet6/exthdr:exthdr -> warning: KLD '/boot/kernel/if_epair.ko' is newer than the linker.hints file
Fatal kernel mode data abort: 'Alignment Fault' on read
trapframe: 0xe014ab00
FSR=00000001, FAR=da87f00e, spsr=20000013
r0 =00000000, r1 =00000001, r2 =00000001, r3 =00000134
r4 =00000000, r5 =00000134, r6 =da87f00e, r7 =da87f022
r8 =00000134, r9 =c0918b04, r10=00000014, r11=e014ac28
r12=00000000, ssp=e014ab90, slr=c04534f4, pc =c048b34c
panic: Fatal abort
cpuid = 1
time = 1691281420
KDB: stack backtrace:
db_trace_self() at db_trace_self
pc = 0xc05ecde4 lr = 0xc0079c70 (db_trace_self_wrapper+0x30)
sp = 0xe014a8b8 fp = 0xe014a9d0
db_trace_self_wrapper() at db_trace_self_wrapper+0x30
pc = 0xc0079c70 lr = 0xc02e99a0 (vpanic+0x140)
sp = 0xe014a9d8 fp = 0xe014a9f8
r4 = 0x00000100 r5 = 0x00000000
r6 = 0xc07597e2 r7 = 0xc0aeaec8
vpanic() at vpanic+0x140
pc = 0xc02e99a0 lr = 0xc02e9780 (doadump)
sp = 0xe014aa00 fp = 0xe014aa04
r4 = 0xe014ab00 r5 = 0x00000013
r6 = 0xda87f00e r7 = 0x00000001
r8 = 0x00000001 r9 = 0xe0773ba0
r10 = 0xda87f00e
doadump() at doadump
pc = 0xc02e9780 lr = 0xc0611184 (abort_align)
sp = 0xe014aa0c fp = 0xe014aa38
r4 = 0xda87f00e r5 = 0xe014aa04
r6 = 0xc02e9780 r10 = 0xe014aa0c
abort_align() at abort_align
pc = 0xc0611184 lr = 0xc06111f8 (abort_align+0x74)
sp = 0xe014aa40 fp = 0xe014aa58
r4 = 0x00000013 r10 = 0xda87f00e
abort_align() at abort_align+0x74
pc = 0xc06111f8 lr = 0xc0610e18 (abort_handler+0x498)
sp = 0xe014aa60 fp = 0xe014aaf8
r4 = 0x00000000 r10 = 0xda87f00e
abort_handler() at abort_handler+0x498
pc = 0xc0610e18 lr = 0xc05ef6ac (exception_exit)
sp = 0xe014ab00 fp = 0xe014ac28
r4 = 0x00000000 r5 = 0x00000134
r6 = 0xda87f00e r7 = 0xda87f022
r8 = 0x00000134 r9 = 0xc0918b04
r10 = 0x00000014
exception_exit() at exception_exit
pc = 0xc05ef6ac lr = 0xc04534f4 (ip_input+0x404)
sp = 0xe014ab90 fp = 0xe014ac28
r0 = 0x00000000 r1 = 0x00000001
r2 = 0x00000001 r3 = 0x00000134
r4 = 0x00000000 r5 = 0x00000134
r6 = 0xda87f00e r7 = 0xda87f022
r8 = 0x00000134 r9 = 0xc0918b04
r10 = 0x00000014 r12 = 0x00000000
udp_input() at udp_input+0x1c0
pc = 0xc048b34c lr = 0xc04534f4 (ip_input+0x404)
sp = 0xe014ac30 fp = 0xe014ac70
r4 = 0x00000001 r5 = 0x00000000
r6 = 0x00000000 r7 = 0x01000193
r8 = 0xda87f00e r9 = 0xc094a938
r10 = 0x00000014
ip_input() at ip_input+0x404
pc = 0xc04534f4 lr = 0xc04235bc (netisr_dispatch_src+0x100)
sp = 0xe014ac78 fp = 0xe014aca0
r4 = 0x00000004 r5 = 0xda854000
r6 = 0x00000000 r7 = 0xc0b5a2f8
r8 = 0x00000000 r9 = 0xc57f7780
r10 = 0x00000008
netisr_dispatch_src() at netisr_dispatch_src+0x100
pc = 0xc04235bc lr = 0xc041a384 (ether_demux+0x1bc)
sp = 0xe014aca8 fp = 0xe014acc0
r4 = 0xda854000 r5 = 0x00000001
r6 = 0xdb846400 r7 = 0x5e4a6f28
r8 = 0x00000000 r9 = 0xc57f7780
r10 = 0x00000008
ether_demux() at ether_demux+0x1bc
pc = 0xc041a384 lr = 0xc041bb68 (ether_nh_input+0x3dc)
sp = 0xe014acc8 fp = 0xe014acf0
r4 = 0xdb846400 r5 = 0xda854000
r6 = 0xda87f000 r10 = 0x00000008
ether_nh_input() at ether_nh_input+0x3dc
pc = 0xc041bb68 lr = 0xc04235bc (netisr_dispatch_src+0x100)
sp = 0xe014acf8 fp = 0xe014ad20
r4 = 0x00000006 r5 = 0xda854000
r6 = 0x00000000 r7 = 0xc0b5a378
r8 = 0x5e4a6f28 r9 = 0xc57f7780
r10 = 0x00000000
netisr_dispatch_src() at netisr_dispatch_src+0x100
pc = 0xc04235bc lr = 0xc041a808 (ether_input+0xec)
sp = 0xe014ad28 fp = 0xe014ad60
r4 = 0xdb846400 r5 = 0x00000000
r6 = 0xda854000 r7 = 0x00000000
r8 = 0x5e4a6f28 r9 = 0xc57f7780
r10 = 0x00000000
ether_input() at ether_input+0xec
pc = 0xc041a808 lr = 0xe098310c ($a.10+0xbc)
sp = 0xe014ad68 fp = 0xe014ad90
r4 = 0xdb846400 r5 = 0xdb7bf8c0
r6 = 0x00000000 r7 = 0xda854000
r8 = 0xe09724d3 r9 = 0xdb7bf8d0
r10 = 0x00000000
$a.10() at $a.10+0xbc
pc = 0xe098310c lr = 0xc03504dc (taskqueue_run_locked+0xb8)
sp = 0xe014ad98 fp = 0xe014ade0
r4 = 0xe0769e00 r5 = 0xe0769e50
r6 = 0xdb7bf8ec r7 = 0x00000001
r8 = 0x00000001 r9 = 0xc0768ff7
r10 = 0x00000000
taskqueue_run_locked() at taskqueue_run_locked+0xb8
pc = 0xc03504dc lr = 0xc0351560 (taskqueue_thread_loop+0x108)
sp = 0xe014ade8 fp = 0xe014ae18
r4 = 0x00000000 r5 = 0xe0769e00
r6 = 0xe0769e40 r7 = 0xc073cb53
r8 = 0xe0769e50 r9 = 0x00000100
r10 = 0xc0afde44
taskqueue_thread_loop() at taskqueue_thread_loop+0x108
pc = 0xc0351560 lr = 0xc02a384c (fork_exit+0xa0)
sp = 0xe014ae20 fp = 0xe014ae38
r4 = 0xe0773ba0 r5 = 0xc0ada560
r6 = 0xc0351458 r7 = 0xe0993f94
r8 = 0xe014ae40 r9 = 0xc0afab7c
fork_exit() at fork_exit+0xa0
pc = 0xc02a384c lr = 0xc05ef640 (swi_exit)
sp = 0xe014ae40 fp = 0x00000000
r4 = 0xc0351458 r5 = 0xe0993f94
r6 = 0xc0942429 r7 = 0xc72f21d0
r8 = 0xc0ada900 r10 = 0xc0afde44
swi_exit() at swi_exit
pc = 0xc05ef640 lr = 0xc05ef640 (swi_exit)
sp = 0xe014ae40 fp = 0x00000000
KDB: enter: panic
I've reported another backtrace previously that
I'll not repeat here. But such was for my personal
build context, unlike here.
"'Alignment Fault' on read" happens even on the cortex-A7
system, no lib32 involved, no chroot involved. The details
may vary for the backtrace across the contexts.
NON-FAILURE EXAMPLE for the cortex-A7 context:
In my in-armv7 chroot on aarch64 and/or lib32 based kyua,
testing I got crashes for the below type of test that I'm
not seeing in this cortex-A7 armv7 context:
# kldload -v -n if_bridge.ko
Loaded if_bridge.ko, id=5
# /usr/bin/kyua test -k /usr/tests/Kyuafile sys/net/if_bridge_test:gif
sys/net/if_bridge_test:gif -> failed: atf-check failed; see the output of the test for details [12.344s]
Results file id is usr_tests.20230806-005112-971198
Results saved to /root/.kyua/store/results.usr_tests.20230806-005112-971198.db
0/1 passed (1 failed)
I'll have to continue to look into it for armv7 chroot on aarch64 and
in the hackish lib32-involved kyua-based testing. The overall status
suggests that for armv7 chroot and/or lib32, if_bridge.ko involvement
leads to panics.
OTHER POINTS beyond the 3 items . . .
GDB HANDLING OF SYSTEM DUMP FOR /var/crash/core.txt.* :
/var/crash/core.txt.0 ends up with gdb generated material that is
basically useless:
generic dumped core - see /var/crash/vmcore.0
Sun Aug 6 00:32:59 UTC 2023
FreeBSD generic 14.0-CURRENT FreeBSD 14.0-CURRENT armv7 1400093 #0 main-n264491-8a5c836b51ce: Thu Aug 3 12:10:56 UTC 2023 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/arm.armv7/sys/GENERIC arm
panic: Fatal abort
GNU gdb (GDB) 13.1 [GDB v13.1 for FreeBSD]
. . .
Reading symbols from /boot/kernel/kernel...
Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...
warning: kld_current_sos: Can't read filename
/wrkdirs/usr/ports/devel/gdb/work-py39/gdb-13.1/gdb/thread.c:1337: internal-error: switch_to_thread: Assertion `thr != NULL' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
----- Backtrace -----
0xca962b ???
0x11880a7 ???
---------------------
/wrkdirs/usr/ports/devel/gdb/work-py39/gdb-13.1/gdb/thread.c:1337: internal-error: switch_to_thread: Assertion `thr != NULL' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Quit this debugging session? (y or n) [answered Y; input not from terminal]
This is a bug, please report it. For instructions, see:
<https://www.gnu.org/software/gdb/bugs/>.
/wrkdirs/usr/ports/devel/gdb/work-py39/gdb-13.1/gdb/thread.c:1337: internal-error: switch_to_thread: Assertion `thr != NULL' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Create a core file of GDB? (y or n) [answered Y; input not from terminal]
Abort trap (core dumped)
NOTE: I ignored the rest of core.txt.0 after seeing the above.
KYUA LACKS INDICATIONS ABOUT PORTS TO INSTALL AND KERNEL MODULES TO PREINSTALL :
Various kyua tests depend on kernel modules that are not automatically loaded.
Various kyua tests depend on various ports, none of which are automatically
installed.
It looks to me like the appropriateness of various such things depends on context,
so that only a subset might be appropriate by default (without extra configuration).
No information about this seems to be present.
I'll note that running kyua inside an armv7 chroot leads to most kernel modules
needing to be preloaded outside the chroot in order for them to be available
to the chroot's kyua run. The armv7 ports need to be installed in the chroot
as well.
What I've done so far may well not be close to an optimal default. I'll not
get into the details of what I've done here.
===
Mark Millard
marklmi at yahoo.com