From nobody Tue Aug 01 08:25:39 2023 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RFSqn1CBHz4py49 for ; Tue, 1 Aug 2023 08:26:03 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic317-22.consmr.mail.gq1.yahoo.com (sonic317-22.consmr.mail.gq1.yahoo.com [98.137.66.148]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RFSqX2Kywz3MRr for ; Tue, 1 Aug 2023 08:25:59 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=teKQulOb; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.66.148 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1690878357; bh=7r2Njw3yZjak5Gg0oYlfu1T1xASa5l9Lv6Wtqli7CCQ=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=teKQulObUujESI60Su4oGCWmeiJy4oxr8leeqIViTZCHSLBFcDw7mj3m5WZToOVzQ2lO5L3HmPPihFjJ/VEjIzuP3VaErvle9ces+TmXbXlNeCEyss8T7QnBWHH8Jbnoh+dC5pTGS2QdsmTg66f+C7cOqWqaYQgDo818z21jMXPrTL847OMJBWxV1LxQmswxR9vO/dWeXI3GA/g3bKuZ04Mamvm74Vqogl9Yjn/ITmbIkAfvzOPaEsjhqSEFJ66mEj/2n9CNUbUvqhXpUW2ahk2QyymsuB29lC8LU9jB+C/s44JirK1Lw9f3XH6r3TPTAFhfNXOF9fwz9b4jmHY90g== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1690878357; bh=13yFiuTbnn5UDIYLbVuxzw9YqY9fE0tBrjqOwlscMCO=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=FfZ9SN/IAO3Qk1NVzQt2z1PHXE9gmgx7AI2ROp3sp3BRTjZPWG0VB0G3txDQLsa/ZO75H2esnwO/nZzVPNq07DI/7203fLqNJvxAWCkc4AK8UXKoBMOweAbrt/bDEK349LPXkKGg2QtvspnTRStQ5D+Mj8YlSDJW3YrLprN40smKSoQzNA1jr06XE3YF3Lz0GB/xfLKgDx8p/07QPCVfNzM4dPWaYipyhOrN2/1Rme6h4WNToj/2R7vFR2ehiUhNAU4P5sK1Lp1Z2x5q79CtOETQRU73ad/MCBsEAXYSdE8Zfi0fBasUhzr5vCYLZLF2ZNSn+k2HJ2P+fNQx1VZAIw== X-YMail-OSG: j22BC5YVM1kahSVwpRpzyZiREhBbSEWznR7TVf2RH4LnhxeSp.x9F.ZOgvFiHZZ X5IHbLtcFGF1JhGdZMHDIKzQiMIK8lF8Dexd970n4nEuo6P2NFqC.FKAOxFOLDvJuGdYUZfDqvWI TYBu5TsZdNOhOV.7zOUgJcUmz5w_wydqDFR6y3nzItpUpb3BMuuvXGxAOn9r1hMOyQQdWCEwtU06 txC4bIQbj13MDXuEr4J8fIKxbR.R1YVLR_Ev2kVBsYFCl1HvwrhXvJxqDOcd7GjcapsRgsfye4PJ aSI8EvlqGjV6F_S5P7NtmhKS_tOX._dDB5byZyt9MtKyJUQBM11bAuziDVLszAvyRK.KCpcARhyb utWTmQl109fSgzD66EpodrW0G8NETdnBQM0uFBsacxSrTll0EeoKy.LSHN44qTcUMg3Ks8U5refC NSz2w1f1EUCCqrMhYTakBkoktlEOjffvnxTclTL08Pb5bcuGAdGzD_FPE18RCTv5xXpEV6kE1i5c YY7JQd.nWuzAxYsgn2Q7CGml2TDsyTFZosEzZtN9KIAROifUZIXwNHpPrF_ajyliUHLsva6sWTV. 5eaWIptdgzroGJLiOGmUXIwnMDPEWHU51FnAXkE8bFDuVGW_t0HsFEaTX8hcrRDM8CitPAgWz0j6 KFwPtBrRlQZn2o823ViNN2K6U143KairyYkWFjUdkQX42FRCkB1ibbrTkObsAcwF5vYoDN3MXBlz l.3z4BRnLnTtmfTupaJ923hXJVtpauo_xKDi5Pc6xNR5gruCaY_a7MKwRPRDQrFfP8EbYgTwmlBn 3smLOS5_gruFqKL0thYmi2oKxNj6vGLECJvfXjWexqa5njDMpOor2JqrfL5qHkTu9Y3W.J4reU5_ MnAABiMj9asBCxw.mnle1skgwMLtc4hSG_VTM1.p..cEg96rP.qkJ0NznOty25x.qAm1k1OH7Wy6 j7KrVMhS2aGSdd1ue9wqXapkyP9wIbUjyFDa9IJks403MA4BaMMxkIqcfQkkrIRubIy4ojYpMN1a lu8eUz4biy_0cR0U96LcHlsFOMFFJy0TvFDvaYZu.nF3zLSBDTM4ZtdC_vi7273j5SHYELjTjlyS lClMxVs6wlHbbHXHA6LSt3k8lLPM67v.AOcmex3Ofhkvs4A4xCGVgKobNfninTUUjVNHaKEiJyKr hNY3.5YDP.JSQKKlrveWTRQNsTnWjHrN_tC8tjXA9xMygeFqfwwTh3IfZ5QegwLp65xlsbCLi05k tJWNURVCvbyfbF7lFVyQEoH1ZWtX2O5rALBixaz_RcwQLhTDhmYFgeI4ypAgUjtbk0lbQijHmUOU T0hNmyktKjmIkvHc1USi7ioRzCgTsQv0dndKn0cZH04JEFodTPkyYS4njpzh6pbDYW8rlzap_oaL 9kO63sD5bOF5j3vHO.g0DW6bjp9o7DNrnP0R5cYeMTaYQkWdtMwsozbUtCUko4irzOJAHaxUVG26 UpdMOTLuwVf_uxTxFWkP61tPtuKu_XObIPCQKxz6lCVrjB8E9NPGGMwsbpoxAboIETJKkomYIDKE c5vurTpfQn.2sc7cRRL.SWUwJ4.xoouaoHIF5BgsyIt6Ydc0ONwlMnOVvGQYH_RTmG76T6fhp3eo 33YwtGRPANw.kdc222BC9iBmj7IPqXMCelduwQ0ZPUCITDKH0HLxCQlsJ1q.mYCQM8DM.pU_n8.H uqqgYEBY0hJdbSKxJcmsV.7ULDyfXnkLPmJJxnqJKUau3t2lqv9U08ttMK5HA_IJi5gNKcaN3mNs 3cCEbSw0hJ6i6NlZ1wE4GR609P_kTYkuYNfM1BCYLSB6qVw1Pps4B29QBlFcU81nVBYe_92HrFPJ 5idke.lNhlUhOWudpBPLhDA7j24AdZ5U3YGAofsB7jXUClZzA9LhMboaLE6JDEfR1jMnuXR1xX_G vLtij3DvJLrq2iv8vGmQowSjaeUGwJ6w0dI1IKfpIYFHK2syOe29fAcKKDOPDMdZVvGHhLP3oSbL 2n_V2SXUSLbnp0U36nkju3CJG.i1yeCERab.OsajwPUErNE9IrJbfvGlRQGmBgwj5.MV.FVImPA1 ypu01mibR7GwxfIqHXgVtZLZmdEqn3T.PfPL.TLGL9bbvSSoISB26esfwaM2B5ob5KwoGu2pbPtu nfFtRVcjLv6eKJZzgIqdYh7IxxdW2bYTMg5S5SZSHOesf6wXZr9C2agSunbT6fgWtpumgVHwjwzg mBNWMPoOEC90OrlzFTJjQ_uydSJECuq7qwvMPRfdReQsFb7fET432.hJxQfBAGIX3Dx6Use6Krg- - X-Sonic-MF: X-Sonic-ID: b810b3a2-32a4-4997-bef8-3fb0fed15858 Received: from sonic.gate.mail.ne1.yahoo.com by sonic317.consmr.mail.gq1.yahoo.com with HTTP; Tue, 1 Aug 2023 08:25:57 +0000 Received: by hermes--production-bf1-7c4db57b6-87g95 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 389e984e1a39be2279f616d41f7b46c1; Tue, 01 Aug 2023 08:25:51 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Some results from my crude technique of using (some of) Kyua to test aarch64's lib32 vs. kyua runs in a armv7 chroot on aarch64 Message-Id: <84EFE4FC-D8E0-41C0-AC10-D1DBE2262F6E@yahoo.com> Date: Tue, 1 Aug 2023 01:25:39 -0700 To: FreeBSD ARM List , Current FreeBSD X-Mailer: Apple Mail (2.3731.700.6) References: <84EFE4FC-D8E0-41C0-AC10-D1DBE2262F6E.ref@yahoo.com> X-Spamd-Result: default: False [-3.20 / 15.00]; NEURAL_HAM_SHORT(-1.00)[-0.997]; NEURAL_HAM_MEDIUM(-0.99)[-0.989]; NEURAL_HAM_LONG(-0.71)[-0.710]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; RCVD_IN_DNSWL_NONE(0.00)[98.137.66.148:from]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MLMMJ_DEST(0.00)[freebsd-arm@freebsd.org]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.66.148:from] X-Rspamd-Queue-Id: 4RFSqX2Kywz3MRr X-Spamd-Bar: --- [I first report how I tested before reporting on errors that look to be valid kyua reports of issues.] [This testing is with a line commented out in order to prevent sys/net/if_bridge_test:gif from being tested, because of its leading to a panic for lib32 style testing.] I have /usr/obj/DESTDIRs/main-CA7-chroot/ containing an armv7 installworld distrib-dirs distribution DB_FROM_SRC=3D1 result. It also has various ports installed that kyua runs use. I use this tree for both the lib32 and the chroot testing. I have a script to preload various kernel modules: # grep kldload ~/prekyua-kldloads.sh | sort #kldload -v -n ipfw.ko #kldload -v -n pflog.ko #kldload -v -n pfsync.ko kldload -v -n bridgestp.ko kldload -v -n carp.ko kldload -v -n cryptodev.ko kldload -v -n dtrace.ko kldload -v -n dummynet.ko kldload -v -n fdescfs.ko kldload -v -n filemon.ko kldload -v -n geom_concat.ko kldload -v -n geom_eli.ko kldload -v -n geom_gate.ko kldload -v -n geom_mirror.ko kldload -v -n geom_multipath.ko kldload -v -n geom_nop.ko kldload -v -n geom_raid3.ko kldload -v -n geom_shsec.ko kldload -v -n geom_stripe.ko kldload -v -n geom_uzip.ko kldload -v -n if_bridge.ko kldload -v -n if_epair.ko kldload -v -n if_gif.ko kldload -v -n if_infiniband.ko kldload -v -n if_lagg.ko kldload -v -n if_ovpn.ko kldload -v -n if_stf.ko kldload -v -n if_tuntap.ko kldload -v -n if_wg.ko kldload -v -n ipdivert.ko kldload -v -n ipsec.ko kldload -v -n mqueuefs.ko kldload -v -n netgraph.ko kldload -v -n nfsd.ko kldload -v -n ng_bridge.ko kldload -v -n ng_ether.ko kldload -v -n ng_hub.ko kldload -v -n ng_socket.ko kldload -v -n ng_vlan_rotate.ko kldload -v -n nullfs.ko kldload -v -n opensolaris.ko kldload -v -n pf.ko kldload -v -n sctp.ko kldload -v -n sdt.ko kldload -v -n tarfs.ko kldload -v -n tcpmd5.ko kldload -v -n xz.ko kldload -v -n zfs.ko (Some I've listed despite there being built into the kernel or already being loaded for my normal environment.) Likely I'll end up adding some more later. I have some ports used by kyua runs that I build and then install into /usr/obj/DESTDIRs/main-CA7-chroot/ : # more ~/origins/kyua-origins.txt=20 archivers/gtar devel/py-pytest devel/py-pytest-twisted devel/py-twisted lang/perl5.32 lang/python net/scapy security/openvpn security/sudo shells/ksh93 shells/bash sysutils/coreutils sysutils/sg3_utils textproc/jq Likely I'll add some more later. The above, of course, lead to other installs as well. For lib32 testing, I try to control where most *.so* 's that are not based full path references are found. This is via use of LD_32_LIBRARY_PATH . I try to have more programs that are not based on full path references run as armv7 code. This is via use of PATH . So: # env \ LD_32_LIBRARY_PATH=3D/usr/obj/DESTDIRs/main-CA7-chroot/lib\ :/usr/obj/DESTDIRs/main-CA7-chroot/usr/lib\ :/usr/obj/DESTDIRs/main-CA7-chroot/usr/tests/libexec/rtld-elf\ :/usr/obj/DESTDIRs/main-CA7-chroot/usr/tests/lib/libxo\ :/usr/obj/DESTDIRs/main-CA7-chroot/usr/tests/lib/csu/dynamiclib\ :/usr/obj/DESTDIRs/main-CA7-chroot/usr/tests/lib/libc/tls\ :/usr/obj/DESTDIRs/main-CA7-chroot/usr/tests/lib/libc/stdlib\ :/usr/obj/DESTDIRs/main-CA7-chroot/usr/tests/lib/libthr/dlopen\ :/usr/obj/DESTDIRs/main-CA7-chroot/usr/local/lib\ = :/usr/obj/DESTDIRs/main-CA7-chroot/usr/local/lib/python3.9/site-packages\ :/usr/obj/DESTDIRs/main-CA7-chroot/usr/local/lib/python3.9/lib-dynload\ :/usr/obj/DESTDIRs/main-CA7-chroot/usr/local/lib/perl5/5.32/mach/CORE\ :/usr/obj/DESTDIRs/main-CA7-chroot/usr/local/lib/perl5/5.32/mach/auto \ PATH=3D/usr/obj/DESTDIRs/main-CA7-chroot/sbin\ :/usr/obj/DESTDIRs/main-CA7-chroot/bin\ :/usr/obj/DESTDIRs/main-CA7-chroot/usr/sbin\ :/usr/obj/DESTDIRs/main-CA7-chroot/usr/bin\ :/usr/obj/DESTDIRs/main-CA7-chroot/usr/local/sbin\ :/usr/obj/DESTDIRs/main-CA7-chroot/usr/local/bin\ :/usr/obj/DESTDIRs/main-CA7-chroot/root/bin \ /usr/obj/DESTDIRs/main-CA7-chroot/usr/bin/kyua test \ -k /usr/obj/DESTDIRs/main-CA7-chroot/usr/tests/Kyuafile On the Windows Dev Kit 2023 I end up with the lib32 summary being (so far): # kyua report --verbose \ = --results-file=3Dusr_obj_DESTDIRs_main-CA7-chroot_usr_tests.20230731-08082= 0-275974 \ 2>&1 \ | tail -6 =3D=3D=3D> Summary Results read from = /usr/home/root/.kyua/store/results.usr_obj_DESTDIRs_main-CA7-chroot_usr_te= sts.20230731-080820-275974.db Test cases: 8704 total, 1442 skipped, 37 expected failures, 46 broken, = 746 failed Start time: 2023-07-31T08:08:20.858437Z End time: 2023-07-31T10:18:37.393732Z Total time: 6954.365s Of course, some tests labeled as broken/failed are just from the limitations of the techniques involved lib32 based kyua testing. For example the 127 "failures": In the chroot it is currently: # kyua report --verbose \ --results-file=3Dusr_tests.20230731-163737-720329 \ 2>&1 \ | tail -6 =3D=3D=3D> Summary Results read from = /usr/home/root/.kyua/store/results.usr_tests.20230731-163737-720329.db Test cases: 8699 total, 1478 skipped, 38 expected failures, 200 broken, = 664 failed Start time: 2023-07-31T16:37:38.302188Z End time: 2023-08-01T06:57:18.663231Z Total time: 50619.428s To generate files to diff lib32 vs. chroot results I used for lib32: # kyua report --results-filter "" \ = --results-file=3Dusr_obj_DESTDIRs_main-CA7-chroot_usr_tests.20230731-08082= 0-275974 \ 2>&1 \ | grep " -> " \ | sed -e 's@\(.*\) [\[0-9.s\]*]$@\1@' \ | sort -u > ~/kyua_lib32_aarch64_armv7.txt NOTE: That strips off the [TIME-IN-SECONDS] suffix. For inside the chroot I used: # kyua report --results-filter "" \ --results-file=3Dusr_tests.20230731-163737-720329 \ 2>&1 \ | grep " -> " \ | sed -e 's@\(.*\) [\[0-9.s\]*]$@\1@' \ | sort -u > ~/kyua_chroot_aarch64_armv7.txt NOTE: That strips off the [TIME-IN-SECONDS] suffix. Then I did the diff (from outside the chroot): # diff -u ~/kyua_lib32_aarch64_armv7.txt \ /usr/obj/DESTDIRs/main-CA7-chroot/root/kyua_chroot_aarch64_armv7.txt \ | grep "^[+-]" \ | sort -k1.2 -k1.1 > ~/kyua_aarch64_lib32_chroot_armv7_diff.txt I later swapped the first 2 lines to get the -then+ order there as well. See the 1021 lines at: https://gist.github.com/markmi/37400c797bc94e5670622bb36919964e There are interesting differences, such as many sys/net* tests for the chroot context hanging up and timing out after 300s but lib32 having many of these tests pass instead. There are other types of tests that chroot generally has the passing status and lib32 has the failure/broken status. One thing I've noticed is that for lib32 some of the failures trace back to examples of "Inappropriate ioctl for device" or other ioctl behavior differences. I sent out a prior notice to the lists for that. There are examples of -/+ line pairs that only exist because of /tmp/* paths that are different in the text of the 2 lines. So the overall line count overstates the differences by some amount. =3D=3D=3D Mark Millard marklmi at yahoo.com