Re: For snapshot builds: armv7 chroot on aarch64 has kyua test -k /usr/tests/Kyuafile sys/kern/kern_copyin hung up [in getpid?], unkillable, prevents reboot
Date: Thu, 06 Jul 2023 06:42:08 UTC
On Jun 25, 2023, at 17:16, Mark Millard <marklmi@yahoo.com> wrote:
> Using the likes of:
>
> FreeBSD-14.0-CURRENT-arm64-aarch64-ROCK64-20230622-b95d2237af40-263748.img
> and:
> FreeBSD-14.0-CURRENT-arm-armv7-GENERICSD-20230622-b95d2237af40-263748.img
>
> I have shown the following behavior after setting up storage
> media based on them. (This was a test that my builds were not
> odd for the issue.)
>
> Boot the aarch64 media and log in. (Note: I logged in
> as root.)
>
> mount the armv7 media (-noatime is just my habit)
> and then put it to use:
>
> # mount -onoatime /dev/da1s2a /mnt
>
> # chroot /mnt/
>
> # kyua test -k /usr/tests/Kyuafile sys/kern/kern_copyin
> sys/kern/kern_copyin:kern_copyin ->
>
> On the serial console:
>
> # ps -xu
> USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
> root 11 1498.4 0.0 0 256 - RNL 23:24 542:52.92 [idle]
> root 1174 100.0 0.0 0 16 - Rs 23:37 0:00.00 /usr/tests/sys/kern/kern_copyin -vunprivileged-user=tests -r/tmp/kyua.9YUttj/2/result.atf kern_copyin
> root 0 0.0 0.0 0 1616 - DLs 23:24 0:00.50 [kernel]
> root 1 0.0 0.0 11704 1288 - ILs 23:24 0:00.02 /sbin/init
> root 2 0.0 0.0 0 256 - WL 23:24 0:00.26 [clock]
> root 3 0.0 0.0 0 272 - DL 23:24 0:00.00 [crypto]
> root 4 0.0 0.0 0 80 - DL 23:24 0:00.95 [cam]
> root 5 0.0 0.0 0 16 - DL 23:24 0:00.00 [busdma]
> root 6 0.0 0.0 0 16 - DL 23:24 0:00.03 [rand_harvestq]
> root 7 0.0 0.0 0 48 - DL 23:24 0:00.06 [pagedaemon]
> root 8 0.0 0.0 0 16 - DL 23:24 0:00.00 [vmdaemon]
> root 9 0.0 0.0 0 160 - DL 23:24 0:00.38 [bufdaemon]
> root 10 0.0 0.0 0 16 - DL 23:24 0:00.00 [audit]
> root 12 0.0 0.0 0 880 - WL 23:24 0:11.81 [intr]
> root 13 0.0 0.0 0 48 - DL 23:24 0:00.04 [geom]
> root 14 0.0 0.0 0 16 - DL 23:24 0:00.00 [sequencer 00]
> root 15 0.0 0.0 0 160 - DL 23:24 0:06.42 [usb]
> root 16 0.0 0.0 0 16 - DL 23:24 0:00.10 [acpi_thermal]
> root 17 0.0 0.0 0 16 - DL 23:24 0:00.00 [acpi_cooling0]
> root 18 0.0 0.0 0 16 - DL 23:24 0:00.04 [syncer]
> root 19 0.0 0.0 0 16 - DL 23:24 0:00.00 [vnlru]
> root 671 0.0 0.0 13260 2600 - Is 23:25 0:00.00 dhclient: system.syslog (dhclient)
> root 674 0.0 0.0 13260 2752 - Is 23:25 0:00.00 dhclient: dpni0 [priv] (dhclient)
> root 761 0.0 0.0 14572 3972 - Ss 23:25 0:00.02 /sbin/devd
> root 964 0.0 0.0 12832 2764 - Is 23:25 0:00.02 /usr/sbin/syslogd -s
> root 1033 0.0 0.0 13012 2604 - Ss 23:25 0:00.01 /usr/sbin/cron -s
> root 1058 0.0 0.0 21052 8308 - Is 23:25 0:00.01 sshd: /usr/sbin/sshd [listener] 0 of 10-100 startups (sshd)
> root 1078 0.0 0.0 21288 9304 - Is 23:26 0:00.09 sshd: root@pts/0 (sshd)
> root 1175 0.0 0.0 21288 9496 - Is 23:37 0:00.04 sshd: root@pts/1 (sshd)
> root 1074 0.0 0.0 13380 3008 u0 Is 23:25 0:00.01 login [pam] (login)
> root 1075 0.0 0.0 13460 3292 u0 S 23:25 0:00.02 -sh (sh)
> root 1233 0.0 0.0 13588 3016 u0 R+ 00:00 0:00.00 ps -xu
> root 1081 0.0 0.0 13460 3328 0 Is 23:26 0:00.02 -sh (sh)
> root 1170 0.0 0.0 5788 2884 0 I 23:36 0:00.02 /bin/sh -i
> root 1172 0.0 0.0 10408 7192 0 I+ 23:37 0:00.30 kyua test -k /usr/tests/Kyuafile sys/kern/kern_copyin
> root 1178 0.0 0.0 13460 3320 1 Is+ 23:38 0:00.01 -sh (sh)
>
> 1174 is stuck, even if one waits for 30min+.
> kill and kill -9 will not kill 1174.
>
> "shutdown -r now" hangs before the reboot happens
> and reports: "some processes would not die".
>
> An interesting property is that ps and top disagree
> about 1174 CPU usage: ps 100%, top 0%. But top also
> indicates 1174 always has CPU0 "STATE". (Across
> tests CPUn varies but within a test it has
> a fixed n.)
>
> I have also seen ps "STAT" being RXs.
>
> The following is from my earlier activity with my own
> builds involved, here 1119, not the 1174 from above.
> truss reports as the last thing for the stuck process
> as "getpid()".
>
> . . .
> 1119: 0.588983953 fstatat(AT_FDCWD,"/usr/tests/sys/kern/kern_copyin",{ mode=-r-xr-xr-x ,inode=111756,size=9776,blksize=10240 },AT_SYMLINK_NOFOLLOW) = 0 (0x0)
> 1119: 0.589065030 mmap(0x0,20480,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 1074188288 (0x4006d000)
> 1119: 0.589227544 openat(AT_FDCWD,"/tmp/kyua.aBQv6E/2/result.atf",O_WRONLY|O_CREAT|O_TRUNC,0644) = 3 (0x3)
> 1119: 0.589276503 getpid() = 1119 (0x45f)
>
>
>
> For reference, from inside an armv7 chroot session
> before doing such a test:
>
> # uname -apKU
> FreeBSD generic 14.0-CURRENT FreeBSD 14.0-CURRENT #0 main-n263748-b95d2237af40: Thu Jun 22 11:10:50 UTC 2023 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC arm armv7 1400090 1400090
I've replicated the same sort of hangup based on:
aarch64 (booted):
# uname -apKU
FreeBSD CA72-16Gp-ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #0 n263893-0631830a7a3c-dirty: Wed Jul 5 13:54:15 PDT 2023 root@CA72-16Gp-ZFS:/usr/obj/BUILDs/alt-main-CA72-nodbg-clang-alt/usr/alt-main-src/arm64.aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64 1400092 1400092
armv7 (as seen in a chroot use):
# uname -apKU
FreeBSD CA72-16Gp-ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #0 n263893-0631830a7a3c-dirty: Wed Jul 5 13:54:15 PDT 2023 root@CA72-16Gp-ZFS:/usr/obj/BUILDs/alt-main-CA72-nodbg-clang-alt/usr/alt-main-src/arm64.aarch64/sys/GENERIC-NODBG-CA72 arm armv7 1400092 1400092
===
Mark Millard
marklmi at yahoo.com