Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

1.46.0 fails to build with llvm-libunwind enabled #76020

Closed
Keruspe opened this issue Aug 28, 2020 · 31 comments
Closed

1.46.0 fails to build with llvm-libunwind enabled #76020

Keruspe opened this issue Aug 28, 2020 · 31 comments
Labels
C-bug Category: This is a bug.

Comments

@Keruspe
Copy link
Contributor

Keruspe commented Aug 28, 2020

On one of my systems which is entirely based on llvm/clang/libc++/llvm-libunwind 1.46.0 builds fine, but on the other "traditional" ones, it fails with

error: linking with `/usr/host/bin/x86_64-pc-linux-gnu-cc` failed: exit code: 1                                                                                                                                                                                                                                                
  |                                                                                                                                                                                                                                                                                                                            
  = note: "/usr/host/bin/x86_64-pc-linux-gnu-cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib64/rustlib/x86_64-unknown-linux-gnu/lib" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0
-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/rustc_binary-7b7ea85adaabb772.rustc_binary.4nmnfy3t-cgu.0.rcgu.o" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/rustc_binary-7b7e
a85adaabb772.rustc_binary.4nmnfy3t-cgu.1.rcgu.o" "-o" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/rustc_binary-7b7ea85adaabb772" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro" "-Wl,-znow" "-Wl,-O1" "-nodefaultlibs" "-L" 
"/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps" "-L" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/release/deps" "-L" "/var/tmp/paludis/build/dev-lang-r
ust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/build/jemalloc-sys-741a979a41c097fb/out/build/lib" "-L" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/build/psm
-434df6e9150bc0ba/out" "-L" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/build/rustc_llvm-37977d896f0c1d8a/out" "-L" "/usr/x86_64-pc-linux-gnu/lib/llvm/10/lib" "-L" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc
-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib64/rustlib/x86_64-unknown-linux-gnu/lib" "-L" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps" "-lrustc_driver-ca92058eee3505a8" "-Wl,-Bstatic" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/libjemalloc_sys-40622d79400e9af9.rlib" "-Wl,--start-group" "-L" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib64/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bdynamic" "-lstd-230cc113f5bd99f4" "-Wl,--end-group" "-Wl,-Bstatic" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib64/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-5c27ddf29299677e.rlib" "-Wl,-Bdynamic" "-lz" "-lrt" "-ldl" "-ltinfo" "-lpthread" "-lm" "-lxml2" "-lstdc++" "-lpthread" "-lutil" "-ldl" "-lutil" "-ldl" "-lrt" "-lpthread" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-ldl" "-lutil"                                                                                                                              = note: /usr/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-ld: /var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-ca92058eee3505a8.so: undefined reference to `_Unwind_Resume'                                
          /usr/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-ld: /var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-ca92058eee3505a8.so: undefined reference to `_Unwind_GetIP'                                 
          /usr/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-ld: /var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-ca92058eee3505a8.so: undefined reference to `_Unwind_Backtrace'                             
          collect2: error: ld returned 1 exit status      

1.45.2 compiles fine.

I'm currently doing some testing but I suspect that 21abc88 is the culprit

@Keruspe Keruspe added the C-bug Category: This is a bug. label Aug 28, 2020
@Keruspe
Copy link
Contributor Author

Keruspe commented Aug 28, 2020

I can confirm that reverting #72746 fixes the build

@tmandry
Copy link
Member

tmandry commented Sep 3, 2020

Can you provide more details about how you're building Rust? Are you using out-of-tree LLVM?

@Keruspe
Copy link
Contributor Author

Keruspe commented Sep 4, 2020

I package rust for Exherbo (source-based) and I build it using its package manager.
For stable rust, we use the system llvm by default indeed

@Keruspe
Copy link
Contributor Author

Keruspe commented Sep 23, 2020

@tmandry How can we get this to move forward?

I don't know how to reproduce the initial issue to find another fix but the current one is either wrong on incomplete given this failure

@petrhosek
Copy link
Contributor

petrhosek commented Oct 8, 2020

The use case that was addressed by #72746 is mixing C++ and Rust code in a single process. In those cases, we don't want libunwind that's linked into Rust binaries to preempt libunwind symbols of C++ binaries because the two unwinders may be incompatible. This is particularly problematic when libunwind used by C++ is sanitized but Rust one isn't which leads to runtime failure that we observed in Fuchsia.

The solution is to build libunwind as a hermetic library to avoid re-exporting its symbols. We already use -fvisibility=hidden when building libunwind but that doesn't take effect unless we disable visibility annotations which is what #72746 does.

I haven't managed to reproduce this failure locally, but my guess is that in your build librustc_driver doesn't explicitly link unwinder and instead relies on unwinder symbols being provided transitively by some other library. After #72746, those symbols are no longer exported hence the link failure. If that's indeed the case, then I think the correct solution isn't to revert that change but to find out why unwinder isn't being linked explicitly.

@Keruspe
Copy link
Contributor Author

Keruspe commented Oct 8, 2020

Could this be somehow related to the use of -Wl,--as-needed?

@Keruspe
Copy link
Contributor Author

Keruspe commented Oct 8, 2020

It is not, just retries disabling it.

The failing linker invocation is:

"/usr/host/bin/x86_64-pc-linux-gnu-cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-Wl,--eh-frame-hdr" "-L" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib64/rustlib/x86_64-unknown-linux-gnu/l
ib" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/rustc_binary-43d4d556549614b1.rustc_binary.98yjbizw-cgu.0.rcgu.o" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc
-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/rustc_binary-43d4d556549614b1.rustc_binary.98yjbizw-cgu.1.rcgu.o" "-o" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rust
c/x86_64-unknown-linux-gnu/release/deps/rustc_binary-43d4d556549614b1" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro" "-Wl,-znow" "-Wl,-O1" "-nodefaultlibs" "-L" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-
unknown-linux-gnu/release/deps" "-L" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/release/deps" "-L" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-ru
stc/x86_64-unknown-linux-gnu/release/build/jemalloc-sys-a57df08f1cff1043/out/build/lib" "-L" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/build/psm-4c77fdac541dea43/out" "-L" 
"/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/build/rustc_llvm-3d5b9a19a4407b23/out" "-L" "/usr/x86_64-pc-linux-gnu/lib/llvm/10/lib" "-L" "/var/tmp/paludis/build/dev-lang-rust-
1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib64/rustlib/x86_64-unknown-linux-gnu/lib" "-L" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps" 
"-lrustc_driver-45869c97924aea2d" "-Wl,-Bstatic" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/libjemalloc_sys-10e304a877f3c25b.rlib" "-Wl,--start-group" "-L" "/var/tmp/pa
ludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib64/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bdynamic" "-lstd-9779a050e626c3f3" "-Wl,--end-group" "-Wl,-Bstatic" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc
-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib64/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-6758141eee354fd1.rlib" "-Wl,-Bdynamic" "-lz" "-lrt" "-ldl" "-ltinfo" "-lpthread" "-lm" "-lxml2" "-lstdc++" "-lpthread" "-lutil" "-ldl" "-lutil" "-ldl" "
-lrt" "-lpthread" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-ldl" "-lutil"

Looks like rustc adds some --as-needed even if it's not in LDFLAGS.

Let's get the interesting part out of this:

"-lstd-9779a050e626c3f3" "-lz" "-lrt" "-ldl" "-ltinfo" "-lpthread" "-lm" "-lxml2" "-lstdc++" "-lpthread" "-lutil" "-ldl" "-lutil" "-ldl" "-lrt" "-lpthread" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-ldl" "-lutil"

I don't see any "-lunwind" in the linker arguments, or am I missing something here?

@Keruspe
Copy link
Contributor Author

Keruspe commented Oct 8, 2020

I see in the logs libunwind-32bae9364f36982a.rmeta being mentioned during builds of panic_unwind, and libunwind-32bae9364f36982a.rlib being mentioned during builds of std.
-lunwind never appears.

@petrhosek when and how is this linkage supposed to happen?

@Keruspe
Copy link
Contributor Author

Keruspe commented Oct 8, 2020

Inside libstd.so I see the two "missing" symbols:

00000000000acf70 l     F .text  00000180 _Unwind_Backtrace
00000000000ae280 l     F .text  00000064 _Unwind_GetIP

@mati865
Copy link
Contributor

mati865 commented Oct 8, 2020

@Keruspe can you look for the same symbols in librustc_driver*.so?

@Keruspe
Copy link
Contributor Author

Keruspe commented Oct 8, 2020

0000000000000000         *UND*  00000000 _Unwind_Backtrace
0000000000000000         *UND*  00000000 _Unwind_GetIP

@mati865
Copy link
Contributor

mati865 commented Oct 8, 2020

@petrhosek IIUC before #72746 when rust.llvm-libunwind in the config.toml was enabled, whole libunwind was reexported by libstd-<hash>.so (this option disables linking unwind library from the system) which is not essentially correct.
We reexport necessary symbols for our use here so librustc_driver_<hash>.so should have those symbols provided by libstd-<hash>.so.
Does this sound right?

On one of my systems which is entirely based on llvm/clang/libc++/llvm-libunwind 1.46.0 builds fine, but on the other "traditional" ones, it fails

I haven't managed to reproduce this failure locally

My gut says it's BFD vs LLD difference but I cannot test it soon.

@Keruspe
Copy link
Contributor Author

Keruspe commented Oct 8, 2020

Testing with lld

@Keruspe
Copy link
Contributor Author

Keruspe commented Oct 8, 2020

Same error with lld and bfd.

I've come to the same understanding as you, fwiw. And since rustc_driver doesn't list the unwind crate in its dependencies, I fail to see how it could get the symbols the same way std does.

Maybe when building rust with the system llvm and with llvm-libunwind enabled we should dynamically link to the system llvm-libunwind instead?

@Keruspe
Copy link
Contributor Author

Keruspe commented Oct 8, 2020

#77703 fixes the build for me. Also, since I was using the system llvm, using the system llvm-libunwind makes sense too I think

@Keruspe
Copy link
Contributor Author

Keruspe commented Oct 11, 2020

@petrhosek @mati865 do you suggest adding the unwind crate as a dependency for rustc_driver to properly fix the issue with in-tree llvm-libunwind?

When attempting to do that, I ended up with errors about things not being Sized, I guess it's some kind of bootstrapping problem, not sure what to look for to fix this

@Keruspe
Copy link
Contributor Author

Keruspe commented Oct 11, 2020

Btw, simple way to reproduce, copy config.toml.example as config.toml and just set llvm-libunwind to true, then ./x.py build

@mati865
Copy link
Contributor

mati865 commented Oct 11, 2020

do you suggest adding the unwind crate as a dependency for rustc_driver to properly fix the issue with in-tree llvm-libunwind?

That would solve the issue for rustc but everything else that links to libstd.so (tools, tests, ...) would fail.

I'd be useful to know how Fuchsia differs from Linux here since this supposedly works there.

I personally don't have time to investigate this issue.

Keruspe added a commit to Keruspe/rust that referenced this issue Oct 12, 2020
Fixes rust-lang#76020

Signed-off-by: Marc-Antoine Perennou <Marc-Antoine@Perennou.com>
@petrhosek
Copy link
Contributor

I think I understand the problem is now. We build both static and shared version libstd, but should be only using the hermetic unwinder for the static version, for shared version we should be using the non-hermetic one because the shared library is expected to re-export unwinder API.

@mati865
Copy link
Contributor

mati865 commented Oct 13, 2020

Sounds plausible to me.

JohnTitor added a commit to JohnTitor/rust that referenced this issue Oct 26, 2020
…mulacrum

add system-llvm-libunwind config option

allows using the system-wide llvm-libunwind as the unwinder

Workaround for rust-lang#76020
@John-Nagle
Copy link

Bug seems to be back. See #77394

@jyn514
Copy link
Member

jyn514 commented Mar 12, 2022

cc @jonhoo, this might be related to #94719

@jonhoo
Copy link
Contributor

jonhoo commented Mar 12, 2022

Hmm, that's 1.59.0, which doesn't have any of my changes around linking in it. Though it could be they my change will fix it I suppose 😅

@John-Nagle
Copy link

Try this on Linux with your version, to check, please.

git clone https://github.com/BVE-Reborn/rend3.git 
cd rend3
cargo build --target i686-pc-windows-gnu

This will build on Linux with a Linux target, but not with a i686-pc-windows-gnu target.

@jonhoo
Copy link
Contributor

jonhoo commented Mar 12, 2022

If you want to test it with my various fixes around linker flags, those should all be active on nightly I think. Or possibly tomorrow's nightly.

@John-Nagle
Copy link

No joy. Just downloaded nightly.

rustc --version
rustc 1.61.0-nightly (335ffbf 2022-03-11)

still fails with the linker unable to find `_Unwind_Resume'.

Full compile listing.

unwindbug01.txt

@jonhoo
Copy link
Contributor

jonhoo commented Mar 12, 2022

I think you may need today's nightly to get my latest fix. Though I still don't know that my changes would actually fix this issue — this doesn't feel related to whether libstdc++ is statically compiled.

@John-Nagle
Copy link

OK. Nightly hasn't updated yet. Still "rustc 1.61.0-nightly (335ffbf 2022-03-11)". I'll try later.

@John-Nagle
Copy link

John-Nagle commented Mar 13, 2022

Nightly has updated to rustc 1.61.0-nightly (f103b29 2022-03-12).

Same problem: undefined reference to _Unwind_Resume'` Compile log attached.
unwindbug02.txt

But I was building for a Windows 32 bit target, which nobody cares about much any more. That doesn't work.
If i build for a Windows 64 bit target, it works. Which is what I really need. Thanks.

@mati865
Copy link
Contributor

mati865 commented Mar 13, 2022

@John-Nagle this issue is for building Rust with specific config option not using it.

Same problem: undefined reference to _Unwind_Resume'`
But I was building for a Windows 32 bit target, which nobody cares about much any more. That doesn't work.

It is tested every time PR is merged.
I think you are building on Ubuntu 20.04 which ships SLJL exceptions rateher than Dwarf for i686. AFAIK this is fixed in newer Ubuntu releases.

@jyn514
Copy link
Member

jyn514 commented Feb 3, 2023

#77703 fixes the build for me. Also, since I was using the system llvm, using the system llvm-libunwind makes sense too I think

I think I understand the problem is now. We build both static and shared version libstd, but should be only using the hermetic unwinder for the static version, for shared version we should be using the non-hermetic one because the shared library is expected to re-export unwinder API.

I'm going to close this since it sounds like the original issue is fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-bug Category: This is a bug.
Projects
None yet
7 participants