Some results from my crude technique of using (some of) Kyua to test aarch64's lib32 vs. kyua runs in a armv7 chroot on aarch64

[I first report how I tested before reporting on errors
that look to be valid kyua reports of issues.]

[This testing is with a line commented out in order to
prevent sys/net/if_bridge_test:gif from being tested,
because of its leading to a panic for lib32 style
testing.]

I have /usr/obj/DESTDIRs/main-CA7-chroot/ containing an
armv7 installworld distrib-dirs distribution DB_FROM_SRC=1
result. It also has various ports installed that kyua
runs use. I use this tree for both the lib32 and the
chroot testing.

I have a script to preload various kernel modules:

# grep kldload ~/prekyua-kldloads.sh | sort
#kldload -v -n ipfw.ko
#kldload -v -n pflog.ko
#kldload -v -n pfsync.ko
kldload -v -n bridgestp.ko
kldload -v -n carp.ko
kldload -v -n cryptodev.ko
kldload -v -n dtrace.ko
kldload -v -n dummynet.ko
kldload -v -n fdescfs.ko
kldload -v -n filemon.ko
kldload -v -n geom_concat.ko
kldload -v -n geom_eli.ko
kldload -v -n geom_gate.ko
kldload -v -n geom_mirror.ko
kldload -v -n geom_multipath.ko
kldload -v -n geom_nop.ko
kldload -v -n geom_raid3.ko
kldload -v -n geom_shsec.ko
kldload -v -n geom_stripe.ko
kldload -v -n geom_uzip.ko
kldload -v -n if_bridge.ko
kldload -v -n if_epair.ko
kldload -v -n if_gif.ko
kldload -v -n if_infiniband.ko
kldload -v -n if_lagg.ko
kldload -v -n if_ovpn.ko
kldload -v -n if_stf.ko
kldload -v -n if_tuntap.ko
kldload -v -n if_wg.ko
kldload -v -n ipdivert.ko
kldload -v -n ipsec.ko
kldload -v -n mqueuefs.ko
kldload -v -n netgraph.ko
kldload -v -n nfsd.ko
kldload -v -n ng_bridge.ko
kldload -v -n ng_ether.ko
kldload -v -n ng_hub.ko
kldload -v -n ng_socket.ko
kldload -v -n ng_vlan_rotate.ko
kldload -v -n nullfs.ko
kldload -v -n opensolaris.ko
kldload -v -n pf.ko
kldload -v -n sctp.ko
kldload -v -n sdt.ko
kldload -v -n tarfs.ko
kldload -v -n tcpmd5.ko
kldload -v -n xz.ko
kldload -v -n zfs.ko

(Some I've listed despite there being built into the
kernel or already being loaded for my normal environment.)

Likely I'll end up adding some more later.

I have some ports used by kyua runs that I build and then
install into /usr/obj/DESTDIRs/main-CA7-chroot/ :

# more ~/origins/kyua-origins.txt 
archivers/gtar
devel/py-pytest
devel/py-pytest-twisted
devel/py-twisted
lang/perl5.32
lang/python
net/scapy
security/openvpn
security/sudo
shells/ksh93
shells/bash
sysutils/coreutils
sysutils/sg3_utils
textproc/jq

Likely I'll add some more later. The above, of course,
lead to other installs as well.

For lib32 testing, I try to control where most *.so* 's
that are not based full path references are found. This
is via use of LD_32_LIBRARY_PATH . I try to have more
programs that are not based on full path references run
as armv7 code. This is via use of PATH . So:

# env \
LD_32_LIBRARY_PATH=/usr/obj/DESTDIRs/main-CA7-chroot/lib\
:/usr/obj/DESTDIRs/main-CA7-chroot/usr/lib\
:/usr/obj/DESTDIRs/main-CA7-chroot/usr/tests/libexec/rtld-elf\
:/usr/obj/DESTDIRs/main-CA7-chroot/usr/tests/lib/libxo\
:/usr/obj/DESTDIRs/main-CA7-chroot/usr/tests/lib/csu/dynamiclib\
:/usr/obj/DESTDIRs/main-CA7-chroot/usr/tests/lib/libc/tls\
:/usr/obj/DESTDIRs/main-CA7-chroot/usr/tests/lib/libc/stdlib\
:/usr/obj/DESTDIRs/main-CA7-chroot/usr/tests/lib/libthr/dlopen\
:/usr/obj/DESTDIRs/main-CA7-chroot/usr/local/lib\
:/usr/obj/DESTDIRs/main-CA7-chroot/usr/local/lib/python3.9/site-packages\
:/usr/obj/DESTDIRs/main-CA7-chroot/usr/local/lib/python3.9/lib-dynload\
:/usr/obj/DESTDIRs/main-CA7-chroot/usr/local/lib/perl5/5.32/mach/CORE\
:/usr/obj/DESTDIRs/main-CA7-chroot/usr/local/lib/perl5/5.32/mach/auto \
PATH=/usr/obj/DESTDIRs/main-CA7-chroot/sbin\
:/usr/obj/DESTDIRs/main-CA7-chroot/bin\
:/usr/obj/DESTDIRs/main-CA7-chroot/usr/sbin\
:/usr/obj/DESTDIRs/main-CA7-chroot/usr/bin\
:/usr/obj/DESTDIRs/main-CA7-chroot/usr/local/sbin\
:/usr/obj/DESTDIRs/main-CA7-chroot/usr/local/bin\
:/usr/obj/DESTDIRs/main-CA7-chroot/root/bin \
/usr/obj/DESTDIRs/main-CA7-chroot/usr/bin/kyua test \
-k /usr/obj/DESTDIRs/main-CA7-chroot/usr/tests/Kyuafile

On the Windows Dev Kit 2023 I end up with the lib32 summary
being (so far):

# kyua report --verbose \
               --results-file=usr_obj_DESTDIRs_main-CA7-chroot_usr_tests.20230731-080820-275974 \
               2>&1 \
| tail -6
===> Summary
Results read from /usr/home/root/.kyua/store/results.usr_obj_DESTDIRs_main-CA7-chroot_usr_tests.20230731-080820-275974.db
Test cases: 8704 total, 1442 skipped, 37 expected failures, 46 broken, 746 failed
Start time: 2023-07-31T08:08:20.858437Z
End time:   2023-07-31T10:18:37.393732Z
Total time: 6954.365s

Of course, some tests labeled as broken/failed are just from the
limitations of the techniques involved lib32 based kyua testing.
For example the 127 "failures":

In the chroot it is currently:

# kyua report --verbose \
--results-file=usr_tests.20230731-163737-720329 \
2>&1 \
| tail -6
===> Summary
Results read from /usr/home/root/.kyua/store/results.usr_tests.20230731-163737-720329.db
Test cases: 8699 total, 1478 skipped, 38 expected failures, 200 broken, 664 failed
Start time: 2023-07-31T16:37:38.302188Z
End time:   2023-08-01T06:57:18.663231Z
Total time: 50619.428s


To generate files to diff lib32 vs. chroot results I
used for lib32:

# kyua report --results-filter "" \
--results-file=usr_obj_DESTDIRs_main-CA7-chroot_usr_tests.20230731-080820-275974 \
2>&1 \
| grep "  ->  " \
| sed -e 's@\(.*\)  [\[0-9.s\]*]$@\1@' \
| sort -u > ~/kyua_lib32_aarch64_armv7.txt

NOTE: That strips off the [TIME-IN-SECONDS] suffix.

For inside the chroot I used:

# kyua report --results-filter "" \
--results-file=usr_tests.20230731-163737-720329 \
2>&1 \
| grep "  ->  " \
| sed -e 's@\(.*\)  [\[0-9.s\]*]$@\1@' \
| sort -u > ~/kyua_chroot_aarch64_armv7.txt

NOTE: That strips off the [TIME-IN-SECONDS] suffix.

Then I did the diff (from outside the chroot):

# diff -u ~/kyua_lib32_aarch64_armv7.txt \
/usr/obj/DESTDIRs/main-CA7-chroot/root/kyua_chroot_aarch64_armv7.txt \
| grep "^[+-]" \
| sort -k1.2 -k1.1 > ~/kyua_aarch64_lib32_chroot_armv7_diff.txt

I later swapped the first 2 lines to get the -then+ order
there as well.

See the 1021 lines at:

https://gist.github.com/markmi/37400c797bc94e5670622bb36919964e

There are interesting differences, such as many sys/net* tests
for the chroot context hanging up and timing out after 300s but
lib32 having many of these tests pass instead. There are other
types of tests that chroot generally has the passing status and
lib32 has the failure/broken status.

One thing I've noticed is that for lib32 some of the failures
trace back to examples of "Inappropriate ioctl for device" or
other ioctl behavior differences. I sent out a prior notice to
the lists for that.

There are examples of -/+ line pairs that only exist because of
/tmp/* paths that are different in the text of the 2 lines.
So the overall line count overstates the differences by some
amount.

===
Mark Millard
marklmi at yahoo.com