Re: Dtrace, Rust, and LLVM13

On Wed, Jun 01, 2022 at 09:15:28PM -0600, Alan Somers wrote:
> The best way to profile a Rust program on FreeBSD is with Brendan
> Gregg's Flamegraph[1].  This is based on dtrace's ustack() function.
> It used to work great.  However, Rust v1.56.0 is based on LLVM13[2]
> and now dtrace can't print user stacks anymore.  For example,
> 
> With Rust 1.55.0
>               libc.so.7`__je_malloc_mutex_prefork+0x124
>               libc.so.7`__je_arena_prefork7+0x73
>               libc.so.7`_malloc_prefork+0x15b
>               libthr.so.3`0x392e4a8c4686
>               libthr.so.3`_fork+0x18
>               test-dad15ed382b075cf`nix::unistd::fork::h358225d652a86eab+0xe
>               test-dad15ed382b075cf`test::test_unistd::test_fork_and_waitpid::hb93c7cdf2b79d680+0x36
>               test-dad15ed382b075cf`test::test_unistd::test_fork_and_waitpid::_$u7b$$u7b$closure$u7d$$u7d$::h329a121974ff9291+0x11
>               test-dad15ed382b075cf`core::ops::function::FnOnce::call_once::h2261827bcba63036+0x11
>               test-dad15ed382b075cf`test::__rust_begin_short_backtrace::hefb7644d11da2ff9+0xa
>               test-dad15ed382b075cf`test::run_test::run_test_inner::_$u7b$$u7b$closure$u7d$$u7d$::hdaa0fb71aac8d97e+0x2f3
>               test-dad15ed382b075cf`std::sys_common::backtrace::__rust_begin_short_backtrace::h8bcc057a546c1087+0xce
>               test-dad15ed382b075cf`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf7d978d08be459d0+0x6a
>               test-dad15ed382b075cf`std::sys::unix::thread::Thread::new::thread_start::h6b52ca0eca213387+0x2b

What are the identifiers at the end of each function?

>               libthr.so.3`0x392e4a8c3a7a
> 
> With Rust 1.56.0
>               libc.so.7`__je_malloc_mutex_prefork+0x124
>               libc.so.7`__je_arena_prefork7+0x73
>               libc.so.7`_malloc_prefork+0x15b
>               libthr.so.3`0x1106cebc6686
>               libthr.so.3`_fork+0x18
>               test-b377ad62cc9e0624`nix::unistd::fork::hbf1ed55b658aa870+0xa
>               0x8
>               0xcccccccccccccccc
> 
> See?  dtrace still prints the C part of the stack, but it only prints
> one or sometimes two frames of the Rust stack.

ustack() unwinds the stack using the frame pointer saved in each stack
frame.  It'll fail to unwind code compiled without a frame pointer,
e.g., if -fomit-frame-pointer is specified to a C/C++ compiler.  For
this and similar reasons, we compile the base system with
-fno-omit-frame-pointer by default.

> I'm not a compiler guy, so I don't know how to fix it.  I don't even
> know if the problem lies in Rust or dtrace.  Would any of you smart
> people be able to help here?  This is a pretty important feature for
> Rust development.

I'd guess that Rust started omitting use of the frame pointer in
generated code.  This commit seems to indicate that that's the case:
https://github.com/rust-lang/rust/pull/48786/commits/5b800c231f45fcd823a3e958bb942cd620ceb3e0
Though, it's rather old.  I'm not sure why the problem appeared only
now.  So maybe the problem is elsewhere, but the commit log also
mentions a -Cforce-frame-pointers flag that you could try.

While it's possible to unwind the stack without using a frame pointer, a
frame pointer makes doing so dead simple.  ustack() does the unwinding
in the kernel, in DTrace probe context, and text addresses are resolved
to symbol names in userspace.  So it's generally desirable to keep the
implementation simple.