Dtrace, Rust, and LLVM13

From: Alan Somers <asomers_at_freebsd.org>
Date: Thu, 02 Jun 2022 03:15:28 UTC
The best way to profile a Rust program on FreeBSD is with Brendan
Gregg's Flamegraph[1].  This is based on dtrace's ustack() function.
It used to work great.  However, Rust v1.56.0 is based on LLVM13[2]
and now dtrace can't print user stacks anymore.  For example,

With Rust 1.55.0
              libc.so.7`__je_malloc_mutex_prefork+0x124
              libc.so.7`__je_arena_prefork7+0x73
              libc.so.7`_malloc_prefork+0x15b
              libthr.so.3`0x392e4a8c4686
              libthr.so.3`_fork+0x18
              test-dad15ed382b075cf`nix::unistd::fork::h358225d652a86eab+0xe
              test-dad15ed382b075cf`test::test_unistd::test_fork_and_waitpid::hb93c7cdf2b79d680+0x36
              test-dad15ed382b075cf`test::test_unistd::test_fork_and_waitpid::_$u7b$$u7b$closure$u7d$$u7d$::h329a121974ff9291+0x11
              test-dad15ed382b075cf`core::ops::function::FnOnce::call_once::h2261827bcba63036+0x11
              test-dad15ed382b075cf`test::__rust_begin_short_backtrace::hefb7644d11da2ff9+0xa
              test-dad15ed382b075cf`test::run_test::run_test_inner::_$u7b$$u7b$closure$u7d$$u7d$::hdaa0fb71aac8d97e+0x2f3
              test-dad15ed382b075cf`std::sys_common::backtrace::__rust_begin_short_backtrace::h8bcc057a546c1087+0xce
              test-dad15ed382b075cf`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf7d978d08be459d0+0x6a
              test-dad15ed382b075cf`std::sys::unix::thread::Thread::new::thread_start::h6b52ca0eca213387+0x2b
              libthr.so.3`0x392e4a8c3a7a

With Rust 1.56.0
              libc.so.7`__je_malloc_mutex_prefork+0x124
              libc.so.7`__je_arena_prefork7+0x73
              libc.so.7`_malloc_prefork+0x15b
              libthr.so.3`0x1106cebc6686
              libthr.so.3`_fork+0x18
              test-b377ad62cc9e0624`nix::unistd::fork::hbf1ed55b658aa870+0xa
              0x8
              0xcccccccccccccccc

See?  dtrace still prints the C part of the stack, but it only prints
one or sometimes two frames of the Rust stack.

I'm not a compiler guy, so I don't know how to fix it.  I don't even
know if the problem lies in Rust or dtrace.  Would any of you smart
people be able to help here?  This is a pretty important feature for
Rust development.

-Alan

[1] https://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html
[2] https://github.com/rust-lang/rust/blob/master/RELEASES.md#version-1560-2021-10-21