Re: git: 6926e2699ae5 - main - arm: Add support for using VFP in kernel [td == curthread failed form of panic for bt in gdb]
- Reply: Mark Millard : "Re: git: 6926e2699ae5 - main - arm: Add support for using VFP in kernel [added new: Called fill_fpregs while the kernel is using the VFP]"
- In reply to: Mark Millard : "Re: git: 6926e2699ae5 - main - arm: Add support for using VFP in kernel [td == curthread failed form of panic for bt in gdb]"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Thu, 16 Feb 2023 08:09:26 UTC
[A very simple program gets the failure under gdb
or lldb of example breakpoints.]
On Feb 15, 2023, at 20:29, Mark Millard <marklmi@yahoo.com> wrote:
> On Feb 15, 2023, at 16:08, Mark Millard <marklmi@yahoo.com> wrote:
>
>> Kornel Dulęba <kd_at_FreeBSD.org> wrote on
>> Date: Sat, 04 Feb 2023 19:22:23 UTC :
>>
>>> The branch main has been updated by kd:
>>>
>>> URL: https://cgit.FreeBSD.org/src/commit/?id=6926e2699ae55080f860488895a2a9aa6e6d9b4d
>>>
>>> commit 6926e2699ae55080f860488895a2a9aa6e6d9b4d
>>> Author: Kornel Dulęba <kd@FreeBSD.org>
>>> AuthorDate: 2023-02-04 12:59:30 +0000
>>> Commit: Kornel Dulęba <kd@FreeBSD.org>
>>> CommitDate: 2023-02-04 19:21:43 +0000
>>>
>>> arm: Add support for using VFP in kernel
>>>
>>> Add missing logic to allow in-kernel VFP usage for ARMv7 NEON.
>>> The implementation is strongly based on arm64 code.
>>> It introduces a family of fpu_kern_* functions to enable the usage
>>> of VFP instructions in kernel.
>>> Apart from that the existing armv7 VFP logic was modified,
>>> taking into account that the state of the VFP registers can now
>>> be modified in the kernel.
>>>
>>> Co-developed by: Wojciech Macek <wma@FreeBSD.org>
>>> Sponsored by: Stormshield
>>> Obtained from: Semihalf
>>> Reviewed by: andrew
>>> Differential Revision: https://reviews.freebsd.org/D37419
>>> ---
>>> lib/libthread_db/arch/arm/libpthread_md.c | 21 ++--
>>> sys/arm/arm/exec_machdep.c | 49 ++++----
>>> sys/arm/arm/machdep.c | 1 +
>>> sys/arm/arm/machdep_kdb.c | 31 ++++-
>>> sys/arm/arm/swtch-v6.S | 8 +-
>>> sys/arm/arm/swtch.S | 8 +-
>>> sys/arm/arm/vfp.c | 182 +++++++++++++++++++++++++++++-
>>> sys/arm/arm/vm_machdep.c | 6 +-
>>> sys/arm/include/fpu.h | 7 ++
>>> sys/arm/include/pcb.h | 5 +
>>> sys/arm/include/reg.h | 12 +-
>>> sys/arm/include/vfp.h | 17 +++
>>> 12 files changed, 293 insertions(+), 54 deletions(-)
>>
>> [This is a somewhat adjusted version of a note replying
>> to a Warner note about a panic someone got during a
>> process coredump that was happening.]
>>
>> Just a possible point, given recent kernel floating
>> point work:
>>
>> I tried to do a typical build and test of some
>> benchmark programs that I sometimes use that involve
>> floating point in some of the programs, some use with
>> multithreading involved. (As FreeBSD and g++ progress
>> I tend to do this once and a while, not as often on
>> armv7 as on aarch64.)
>>
>> On armv7, I now usually get a message about a failure
>> of an internal cross-check, which also leads to the
>> program being stopped early. The messaging from run
>> to run varies what the failure is, but the runs should
>> not vary and should not fail the cross-checks --and
>> previously did not, including when I last tried armv7.
>> (Not recently.)
>>
>> For the specific example failures, the initial serial
>> (single thread) test with float involved works but the
>> following multi-thread test in the same program fails
>> and causes the program to stop when it notices there
>> is a problem. (On occasion the cross-check does does
>> not detect a problem.)
>>
>> The programs that do not test floating point do not
>> fail. (Same algorithm on integral types.) These can
>> involve floating point outside the algorithm
>> benchmarked, but with no multi-threading involved for
>> such and no floating point based cross-checks involved.
>>
>> At this point it is far from obvious to me how I
>> would trackdown the specifics of what leads to the
>> failed cross-checks. But the above is suggestive of
>> there being problems for armv7 handling of saving
>> and restoring floating point context for
>> multi-threading in a process, at least. I've no clue
>> if such are strictly limited to the floating point
>> values that show up vs. if there might be wider
>> memory handling problems that result in the process.
>>
>
> Further runs of the benchmark program show that I also
> get cross-check failures for single-threaded (the first
> way it tests).
>
> But it turns out that, even for single treaded execution
> of the algorithm benchmarked, it is not run on the
> process's initial thread but instead on a created thread.
>
> Turns out that for a debug armv7 kernel (debug is not
> what I normally run) attempting a bt in gdb can lead to
> a kernel panic (td == curthread failed) related to
> floating point handling:
>
> . . .
> (gdb) br serial_kernel_runner
> Breakpoint 1 at 0x1db34: serial_kernel_runner. (6 locations)
> (gdb) br parallel_kernel_runner
> Breakpoint 2 at 0x1b43c: parallel_kernel_runner. (6 locations)
> (gdb) run
> Starting program: /root/acpphint/acpphint_kernelsamplers_main-OPi+2E-2048MiB-threads_4-ILP32-FreeBSD_main_n260797_dc1b8c9a846e_32bit-g++_12_O3lto-libc++-cpulockdown
> . . .
>
> Breakpoint 1, serial_kernel_runner<float, unsigned short> (clock_info=..., laps=3, memry=2, ki=...) at acpphint_kernelrunners.cpp:69
> 69 static auto serial_kernel_runner
> (gdb) bt
> #0 serial_panic: Assertion td == curthread failed at /usr/main-src/sys/arm/arm/exec_machdep.c:103
> cpuid = 3
> time = 1676519530
> KDB: stack backtrace:
> db_trace_self() at db_trace_self
> pc = 0xc05f04a0 lr = 0xc007ab0c (db_trace_self_wrapper+0x30)
> sp = 0xe28ea960 fp = 0xe28eaa78
> db_trace_self_wrapper() at db_trace_self_wrapper+0x30
> pc = 0xc007ab0c lr = 0xc02ddc44 (vpanic+0x140)
> sp = 0xe28eaa80 fp = 0xe28eaaa0
> r4 = 0x00000100 r5 = 0x00000000
> r6 = 0xc0790bb4 r7 = 0xc0b1b930
> vpanic() at vpanic+0x140
> pc = 0xc02ddc44 lr = 0xc02dda28 (dump_savectx)
> sp = 0xe28eaaa8 fp = 0xe28eaaac
> r4 = 0xe28eaad0 r5 = 0xbfbfe150
> r6 = 0xe28eaad0 r7 = 0xc076a096
> r8 = 0xdb8a47f4 r9 = 0x00000016
> r10 = 0x00000040
> dump_savectx() at dump_savectx
> pc = 0xc02dda28 lr = 0xc05f3354 (get_vfpcontext+0xb8)
> sp = 0xe28eaab4 fp = 0xe28eaac8
> get_vfpcontext() at get_vfpcontext+0xb8
> pc = 0xc05f3354 lr = 0xc0611148 (cpu_ptrace+0x38)
> sp = 0xe28eaad0 fp = 0xe28eabe8
> r4 = 0xdb75cba0 r5 = 0xbfbfe150
> cpu_ptrace() at cpu_ptrace+0x38
> pc = 0xc0611148 lr = 0xc0360f4c (kern_ptrace+0x810)
> sp = 0xe28eabf0 fp = 0xe28eac70
> r4 = 0xe583dba0 r5 = 0x00000000
> r6 = 0xdb8a47a8 r10 = 0x00000040
> kern_ptrace() at kern_ptrace+0x810
> pc = 0xc0360f4c lr = 0xc0360550 (sys_ptrace+0x1cc)
> sp = 0xe28eac78 fp = 0xe28eadc0
> r4 = 0xe583de5c r5 = 0xe583dba0
> r6 = 0xbfbfe150 r7 = 0x00000000
> r8 = 0x00000000 r9 = 0xe583de50
> r10 = 0xdb756730
> sys_ptrace() at sys_ptrace+0x1cc
> pc = 0xc0360550 lr = 0xc0613b48 (swi_handler+0x170)
> sp = 0xe28eadc8 fp = 0xe28eae38
> r4 = 0xe583dba0 r5 = 0x00000001
> r6 = 0xc090b220 r7 = 0x00000000
> r8 = 0x00000000 r9 = 0xe583de50
> swi_handler() at swi_handler+0x170
> pc = 0xc0613b48 lr = 0xc05f2d90 (swi_exit)
> sp = 0xe28eae40 fp = 0xbfbfe128
> r4 = 0x00000042 r5 = 0x22e61c20
> r6 = 0xbfbfe150 r7 = 0x0000001a
> r8 = 0x00424124 r9 = 0x00000108
> r10 = 0x00000040
> swi_exit() at swi_exit
> pc = 0xc05f2d90 lr = 0xc05f2d90 (swi_exit)
> sp = 0xe28eae40 fp = 0xbfbfe128
> KDB: enter: panic
> [ thread pid 5438 tid 106943 ]
> Stopped at kdb_enter+0x54: ldrb r15, [r15, r15, ror r15]!
>
> Note: the code was built via g++12 but using libc++,
> not libstdc++.
>
> So I tried the b=program variant that does not tryin to
> lock down which CPUs are used by the threads (a completely
> C++20 standard program variant, not FreeBSD specific for
> its used source code). Failure again . . .
>
> (gdb) br serial_kernel_runner
> Breakpoint 1 at 0x1c1bc: serial_kernel_runner. (6 locations)
> (gdb) br parallel_kernel_runner
> Breakpoint 2 at 0x19ac8: parallel_kernel_runner. (6 locations)
> (gdb) run
> Starting program: /root/acpphint/acpphint_kernelsamplers_main-OPi+2E-2048MiB-threads_4-ILP32-FreeBSD_main_n260797_dc1b8c9a846e_32bit-g++_12_O3lto-libc++
> . . .
> Breakpoint 1, serial_kernel_runner<float, unsigned short> (clock_info=..., laps=3, memry=2, ki=...) at acpphint_kernelrunners.cpp:69
> 69 static auto serial_kernel_runner
> (gdb) bt
> #0 serial_kernel_runner<float, unsigned short> (clock_info=...,panic: Assertion td == curthread failed at /usr/main-src/sys/arm/arm/exec_machdep.c:103
> cpuid = 0
> time = 1676520400
> KDB: stack backtrace:
> db_trace_self() at db_trace_self
> pc = 0xc05f04a0 lr = 0xc007ab0c (db_trace_self_wrapper+0x30)
> sp = 0xe2964960 fp = 0xe2964a78
> db_trace_self_wrapper() at db_trace_self_wrapper+0x30
> pc = 0xc007ab0c lr = 0xc02ddc44 (vpanic+0x140)
> sp = 0xe2964a80 fp = 0xe2964aa0
> r4 = 0x00000100 r5 = 0x00000000
> r6 = 0xc0790bb4 r7 = 0xc0b1b930
> vpanic() at vpanic+0x140
> pc = 0xc02ddc44 lr = 0xc02dda28 (dump_savectx)
> sp = 0xe2964aa8 fp = 0xe2964aac
> r4 = 0xe2964ad0 r5 = 0xbfbfe158
> r6 = 0xe2964ad0 r7 = 0xc076a096
> r8 = 0xdb7a511c r9 = 0x00000016
> r10 = 0x00000040
> dump_savectx() at dump_savectx
> pc = 0xc02dda28 lr = 0xc05f3354 (get_vfpcontext+0xb8)
> sp = 0xe2964ab4 fp = 0xe2964ac8
> get_vfpcontext() at get_vfpcontext+0xb8
> pc = 0xc05f3354 lr = 0xc0611148 (cpu_ptrace+0x38)
> sp = 0xe2964ad0 fp = 0xe2964be8
> r4 = 0xdb7ca3e0 r5 = 0xbfbfe158
> cpu_ptrace() at cpu_ptrace+0x38
> pc = 0xc0611148 lr = 0xc0360f4c (kern_ptrace+0x810)
> sp = 0xe2964bf0 fp = 0xe2964c70
> r4 = 0xdb76fba0 r5 = 0x00000000
> r6 = 0xdb7a50d0 r10 = 0x00000040
> kern_ptrace() at kern_ptrace+0x810
> pc = 0xc0360f4c lr = 0xc0360550 (sys_ptrace+0x1cc)
> sp = 0xe2964c78 fp = 0xe2964dc0
> r4 = 0xdb76fe5c r5 = 0xdb76fba0
> r6 = 0xbfbfe158 r7 = 0x00000000
> r8 = 0x00000000 r9 = 0xdb76fe50
> r10 = 0xdb754000
> sys_ptrace() at sys_ptrace+0x1cc
> pc = 0xc0360550 lr = 0xc0613b48 (swi_handler+0x170)
> sp = 0xe2964dc8 fp = 0xe2964e38
> r4 = 0xdb76fba0 r5 = 0x00000001
> r6 = 0xc090b220 r7 = 0x00000000
> r8 = 0x00000000 r9 = 0xdb76fe50
> swi_handler() at swi_handler+0x170
> pc = 0xc0613b48 lr = 0xc05f2d90 (swi_exit)
> sp = 0xe2964e40 fp = 0xbfbfe130
> r4 = 0x00000042 r5 = 0x22e61c20
> r6 = 0xbfbfe158 r7 = 0x0000001a
> r8 = 0x00424124 r9 = 0x00000108
> r10 = 0x00000040
> swi_exit() at swi_exit
> pc = 0xc05f2d90 lr = 0xc05f2d90 (swi_exit)
> sp = 0xe2964e40 fp = 0xbfbfe130
> KDB: enter: panic
> [ thread pid 1107 tid 100140 ]
> Stopped at kdb_enter+0x54: ldrb r15, [r15, r15, ror r15]!
>
> For reference (whitespace may not have
> been preserved):
>
> void
> get_vfpcontext(struct thread *td, mcontext_vfp_t *vfp)
> {
> struct pcb *pcb;
>
> MPASS(td == curthread);
>
> pcb = td->td_pcb;
> if ((pcb->pcb_fpflags & PCB_FP_STARTED) != 0) {
> critical_enter();
> vfp_store(&pcb->pcb_vfpstate, false);
> critical_exit();
> }
> KASSERT(pcb->pcb_vfpsaved == &pcb->pcb_vfpstate,
> ("Called get_vfpcontext while the kernel is using the VFP"));
> memcpy(vfp->mcv_reg, pcb->pcb_vfpstate.reg,
> sizeof(vfp->mcv_reg));
> vfp->mcv_fpscr = pcb->pcb_vfpstate.fpscr;
> }
>
> Unfortunately the benchmark program is far from being a
> minimalist/simple example.
>
> I'm not sure what FreeBSD might have around that would
> have floating point in use but be simple, and possibly
> standardly available, to see if a simpler context is
> available for analogous testing.
>
The program, an example way to build it such that
it can lead to crashes, and 2 ways to get the
FreeBSD crash with it (native armv7 context):
// # cc -std=c17 -pedantic -g -O3 simple_dbl.c
//
// # gdb a.out
// (gdb) br test
// (gdb) run
// FreeBSD CRASHES
//
// # lldb a.out
// (lldb) br set -F test
// FreeBSD CRASHES
#include <stdlib.h>
_Bool test(double v) {
return v<0.5;
}
int main(void) {
return test(drand48());
}
===
Mark Millard
marklmi at yahoo.com