Re: git: 9a2ae72421cd - main - libthr: switch thread and sleepq memory allocator to crt from libc malloc

From: Mark Johnston <markj_at_freebsd.org>
Date: Tue, 14 Jan 2025 20:42:52 UTC 
On Tue, Jan 14, 2025 at 05:55:15PM +0000, Konstantin Belousov wrote:
> The branch main has been updated by kib:
>
> URL: https://cgit.FreeBSD.org/src/commit/?id=9a2ae72421cd75c741984f63b8c9ee89346a188d
>
> commit 9a2ae72421cd75c741984f63b8c9ee89346a188d
> Author:     Konstantin Belousov <kib@FreeBSD.org>
> AuthorDate: 2025-01-14 09:06:58 +0000
> Commit:     Konstantin Belousov <kib@FreeBSD.org>
> CommitDate: 2025-01-14 17:55:08 +0000
>
>     libthr: switch thread and sleepq memory allocator to crt from libc malloc
>
>     There are more complex interactions between malloc and libthr
>     initialization that can happen if libthr functions are called from ELF
>     object' constructors, before libthr is initialized.  Break the
>     dependencies loop by using the private allocator with controlled init.
>
>     Reported by:    yuri
>     Reviewed by:    markj, olce
>     Sponsored by:   The FreeBSD Foundation
>     MFC after:      1 week
>     Differential revision:  https://reviews.freebsd.org/D48454

I see some startup deadlock when running the googletest regression tests
(/usr/tests/lib/googletest/gmock_main) after this commit.  gdb (which
itself also hangs due to this bug) shows:

(gdb) bt
#0  _umtx_op_err () at /home/markj/sb/main/src/lib/libsys/amd64/_umtx_op_err.S:38
#1  0x000015e1ba96fd2c in __thr_umutex_lock (mtx=0x15e1ba974468, id=100113) at /usr/src/lib/libthr/thread/thr_umtx.c:69
#2  0x000015e1ba966a41 in __thr_calloc (num=1, size=17) at /usr/src/lib/libthr/thread/thr_malloc.c:92
#3  0x000015e1ba969213 in mutex_init (mutex=warning: (Internal error: pc 0x15e1bd5c0240 in read in CU, but not in symtab.)
warning: (Error: pc 0x15e1bd5c0240 in address map, but not in symtab.)
0x15e1bd5c0240, mutex_attr=<optimized out>, calloc_cb=<optimized out>)
    at /usr/src/lib/libthr/thread/thr_mutex.c:291
#4  init_static (thread=0x1dc347e4b008, mutex=mutex@entry=warning: (Internal error: pc 0x15e1bd5c0240 in read in CU, but not in symtab.)
warning: (Error: pc 0x15e1bd5c0240 in address map, but not in symtab.)
0x15e1bd5c0240) at /usr/src/lib/libthr/thread/thr_mutex.c:307
#5  0x000015e1ba967192 in check_and_init_mutex (mutex=warning: (Internal error: pc 0x15e1bd5c0240 in read in CU, but not in symtab.)
warning: (Error: pc 0x15e1bd5c0240 in address map, but not in symtab.)
0x15e1bd5c0240, m=<optimized out>) at /usr/src/lib/libthr/thread/thr_mutex.c:606
#6  __Tthr_mutex_lock (mutex=warning: (Internal error: pc 0x15e1bd5c0240 in read in CU, but not in symtab.)
warning: (Error: pc 0x15e1bd5c0240 in address map, but not in symtab.)
0x15e1bd5c0240) at /usr/src/lib/libthr/thread/thr_mutex.c:751
#7  0x000015e1bd54e714 in std::__1::__libcpp_mutex_lock[abi:se190105](pthread_mutex**) (__m=0x15e1ba974468)
    at /home/markj/sb/main/src/contrib/llvm-project/libcxx/include/__thread/support/pthread.h:95
#8  std::__1::__call_once (flag=@0x15e1bd5c0cf0: 0, arg=0x11, func=0x0) at /home/markj/sb/main/src/contrib/llvm-project/libcxx/src/call_once.cpp:44
#9  0x000015e1bd58a6a3 in std::__1::call_once[abi:se190105]<std::__1::locale::id::__get()::$_0>(std::__1::once_flag&, std::__1::locale::id::__get()::$_0&&) (
    __flag=..., __func=...) at /home/markj/sb/main/src/contrib/llvm-project/libcxx/include/__mutex/once_flag.h:131
#10 std::__1::locale::id::__get (this=0x15e1bd5c0cf0 <std::__1::collate<char>::id>) at /home/markj/sb/main/src/contrib/llvm-project/libcxx/src/locale.cpp:596
#11 std::__1::locale::__imp::install<std::__1::collate<char> > (this=0x15e1bd5c0e30 <std::__1::locale::__imp::classic_locale_imp_>, f=<optimized out>)
    at /home/markj/sb/main/src/contrib/llvm-project/libcxx/src/locale.cpp:153
#12 std::__1::locale::__imp::__imp (this=this@entry=0x15e1bd5c0e30 <std::__1::locale::__imp::classic_locale_imp_>, refs=refs@entry=1)
    at /home/markj/sb/main/src/contrib/llvm-project/libcxx/src/locale.cpp:161
#13 0x000015e1bd58c143 in std::__1::__no_destroy<std::__1::locale::__imp>::__emplace[abi:se190105]<unsigned int>(unsigned int&&) (
    this=0x15e1bd5c0e30 <std::__1::locale::__imp::classic_locale_imp_>, __args=<optimized out>)
    at /home/markj/sb/main/src/contrib/llvm-project/libcxx/include/__utility/no_destroy.h:42
#14 std::__1::locale::classic()::$_0::operator()() const (this=<optimized out>) at /home/markj/sb/main/src/contrib/llvm-project/libcxx/src/locale.cpp:502
#15 std::__1::locale::classic () at /home/markj/sb/main/src/contrib/llvm-project/libcxx/src/locale.cpp:500
#16 0x000015e1bd58ea0b in std::__1::locale::__global () at /home/markj/sb/main/src/contrib/llvm-project/libcxx/src/locale.cpp:509
#17 std::__1::locale::locale (this=0x15e1bd5c0798) at /home/markj/sb/main/src/contrib/llvm-project/libcxx/src/locale.cpp:523
#18 0x000015e1bd55fcf0 in std::__1::basic_streambuf<char, std::__1::char_traits<char> >::basic_streambuf (this=warning: (Internal error: pc 0x15e1bd5c0790 in read in CU, but not in symtab.)
warning: (Error: pc 0x15e1bd5c0790 in address map, but not in symtab.)
0x15e1bd5c0790)
    at /home/markj/sb/main/src/contrib/llvm-project/libcxx/include/streambuf:300
#19 0x000015e1bd56d9b8 in std::__1::__stdinbuf<char>::__stdinbuf (this=warning: (Internal error: pc 0x15e1bd5c0790 in read in CU, but not in symtab.)
warning: (Error: pc 0x15e1bd5c0790 in address map, but not in symtab.)
0x15e1bd5c0790, __fp=0x15e1c095a560, __st=<optimized out>)
    at /home/markj/sb/main/src/contrib/llvm-project/libcxx/src/std_stream.h:72
#20 std::__1::DoIOSInit::DoIOSInit (this=<optimized out>) at /home/markj/sb/main/src/contrib/llvm-project/libcxx/src/iostream.cpp:131
#21 0x000015e1bd56e06d in std::__1::ios_base::Init::Init (this=<optimized out>) at /home/markj/sb/main/src/contrib/llvm-project/libcxx/src/iostream.cpp:166
#22 0x000015e1bd56eef5 in __cxx_global_var_init () at /home/markj/sb/main/src/contrib/llvm-project/libcxx/src/iostream_init.h:2
#23 0x000015e1bd56eef5 in global constructors keyed to 000100 () from /lib/libc++.so.1
#24 0x0000357474de453d in objlist_call_init (list=list@entry=0x15e1b85f0a70, lockstate=lockstate@entry=0x15e1b85f0860)
    at /home/markj/sb/main/src/libexec/rtld-elf/rtld.c:3110
#25 0x0000357474de3219 in _rtld (sp=<optimized out>, exit_proc=0x15e1b85f0ae0, objp=0x15e1b85f0ae8) at /home/markj/sb/main/src/libexec/rtld-elf/rtld.c:974
#26 0x0000357474de0019 in rtld_start () at /home/markj/sb/main/src/libexec/rtld-elf/amd64/rtld_start.S:40