meson x rustc library ordering issue

[Paul Mars]

meson 1.3.1-1 is stuck in -proposed with failing autopkgtests on arm64.

This is failing the same way in debian.

Two tests in the test suite are failing when compiling (apparently for the same reason):

2308s = note: /usr/bin/ld: libcpp.a.p/lib.cpp.o: undefined reference to symbol '__stack_chk_guard@@GLIBC_2.17'
2308s /usr/bin/ld: /lib/ld-linux-aarch64.so.1: error adding symbols: DSO missing from command line
2308s collect2: error: ld returned 1 exit status
2308s
2308s = note: some `extern` functions couldn't be found; some native libraries may need to be installed or havetheir path specified
2308s = note: use the `-l` flag to specify native libraries to link
2308s = note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo (see https://doc.rust-lang.org/cargo/reference/build-scripts.html#cargorustc-link-libkindname)

It looks like this is linked to a newly added or missing `-fstack-protector-all` flag.

Some research on this error lead me to try adding this `-fstack-protected` flag when building on arm64.

See my patch [0] and the package in my ppa [1].

But autopkgtests are still failing [2] and it looks like my flag is ignored. I may have missed something in the meson syntax.

[0] https://launchpadlibrarian.net/710064457/meson_1.3.1-1ubuntu1_1.3.1-1ubuntu2.diff.gz
[1] https://launchpad.net/~upils/+archive/ubuntu/test-ppa/+packages
[2] https://autopkgtest.ubuntu.com/results/autopkgtest-noble-upils-test-ppa/noble/arm64/m/meson/20240119_115804_82043@/log.gz

[Jussi Pakkanen (jpakkane)]

What is the diff file against? It alters a file called debian/patches/3-add-stack-protector-on-arm64.patch, but that is neither in the default upstream nor Ubuntu packaging.

[Paul Mars (upils)]

Sorry, the diff I linked was against the previous version (meson_1.3.1-1ubuntu1) I uploaded in my ppa.

Here is the complete diff.

[Jussi Pakkanen (jpakkane)]

That patch seems like a workaround for the real issue. Why is it needed? Shouldn't the package builder set envvars so that it is used for linking?

[Paul Mars (upils)]

I understand this is a failure in the building of one of the rust tests. I do not know if setting envvars at the package builder level to make one test work would be a good idea, because I suppose it could change the behavior of other tests.

I do not know much about meson so this may indeed not solve the root cause.

[Mate Kukri (mkukri)]

To me this seems like a rust linking issue, the C file compiled with gcc, then linked against a C stub works just fine for me.

Versus rustc tries to be annoyingly clever and calls `cc` to link with `-nodefaultlibs` and tries to manually provide all the required libraries, which makes it promptly fail........

It seems like we should fix rustc instead of meson here, but I'll do more investigation first to see if changing the link args that rustc uses actually helps.

[Mate Kukri (mkukri)]

Yep it is indeed a library ordering issue, `-lc` should always come *after* all C static libs and object files in the link command, but it does not.

It seems that meson doesn't pass `-Clink-arg=-lc` to `rustc` at all by default, and it's rustc itself that emits `-lc` at the wrong location.

A workaround is as follows:
```
dep_libc = declare_dependency(link_args : '-lc')

l = static_library(
  'c_accessing_zlib',
  'c_accessing_zlib.c',
  dependencies: [dep_zlib, dep_libc],
)
```

It is rather unweildly to read, but here's an example of an offending link invocation by rustc:
```
LC_ALL="C" PATH="/usr/lib/rustlib/aarch64-unknown-linux-gnu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin" VSLANG="1033" "cc" "/tmp/rustcqxuAcC/symbols.o" "prog.p/prog.prog.3a9e8efe5d7989d1-cgu.0.rcgu.o" "prog.p/prog.2valzr4aprd78wdd.rcgu.o" "-Wl,--as-needed" "-L" "." "-L" "/usr/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/usr/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd-3f505df5bf7b6973.rlib" "/usr/lib/rustlib/aarch64-unknown-linux-gnu/lib/libpanic_unwind-f7bbfb2a4f0106ba.rlib" "/usr/lib/rustlib/aarch64-unknown-linux-gnu/lib/libobject-799d4aa6228550fd.rlib" "/usr/lib/rustlib/aarch64-unknown-linux-gnu/lib/libmemchr-817c1781430e6007.rlib" "/usr/lib/rustlib/aarch64-unknown-linux-gnu/lib/libaddr2line-ad8936af23d8ece8.rlib" "/usr/lib/rustlib/aarch64-unknown-linux-gnu/lib/libgimli-fa9ccab1157705c6.rlib" "/usr/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_demangle-0f0344f8af442a24.rlib" "/usr/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd_detect-f1d03073262366fd.rlib" "/usr/lib/rustlib/aarch64-unknown-linux-gnu/lib/libhashbrown-d2b728fe429f73fd.rlib" "/usr/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-78f4eeade0e7b3f3.rlib" "/usr/lib/rustlib/aarch64-unknown-linux-gnu/lib/libminiz_oxide-386abc7d4431b549.rlib" "/usr/lib/rustlib/aarch64-unknown-linux-gnu/lib/libadler-f2fbe97df215d47a.rlib" "/usr/lib/rustlib/aarch64-unknown-linux-gnu/lib/libunwind-1b9c021e2652ca95.rlib" "/usr/lib/rustlib/aarch64-unknown-linux-gnu/lib/libcfg_if-a31e7b8703ef1917.rlib" "/usr/lib/rustlib/aarch64-unknown-linux-gnu/lib/liblibc-67315dfc1c311b62.rlib" "/usr/lib/rustlib/aarch64-unknown-linux-gnu/lib/liballoc-6f13ad92e9ef28f1.rlib" "/usr/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_std_workspace_core-4647df6db5bfc191.rlib" "/usr/lib/rustlib/aarch64-unknown-linux-gnu/lib/libcore-7ca28360f44e13c1.rlib" "/usr/lib/rustlib/aarch64-unknown-linux-gnu/lib/libcompiler_builtins-bda26fded9d4fbc7.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/usr/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-o" "prog" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-nodefaultlibs" "libc_accessing_zlib.a" "-lz"
```

Which was produced by the following rustc invocation:
```
rustc -C linker=cc --color=always -C debug-assertions=yes -C overflow-checks=no --crate-type bin -g --crate-name prog --emit dep-info=prog.d --emit link=prog --out-dir prog.p -C metadata=prog@exe -Clink-arg=libc_accessing_zlib.a -Clink-arg=-lz -L. ../prog.rs
```

As a note...

[Jussi Pakkanen (jpakkane)]

> it's rustc itself that emits `-lc` at the wrong location

Then this is a rustc bug and needs to be fixed there, not worked around after the fact by adding hacks on top of hacks on top of hacks.

In either case the patch you posted is incorrect. You don't need a dependency just to inject a link argument and in either case it should not go in the static library, because link arguments are never passed to it directly (as the static linker takes no arguments). A better way would be to create a dep that has the library and the corresponding flag, but even that is not really great. Doing it would mean that every single project that links C and Rust code together needs to copypaste the same workaround. This needs to be fixed properly once so that most developers won't even know that this is a thing that exists.

[Mate Kukri (mkukri)]

@jpakkane As I've also responded on the Debian bug, unfortunately my first attempt at a workaround was indeed failing to address the issue, but as far as I am aware only because there are multiple such failing tests.

I agree that injecting the dependency on the static library is not nice, but the linking was failing because that static library was passed to the linker last. I have noticed that it's dependencies come after it, so injecting `-lc` there did produce the correct command line at least for that one test.

As far as it being a rustc issue, I am only 50/50 because rustc doesn't necessarily guarantee that internally it always is linking against libc, and it also cannot know that external dependencies passed to it are really written in C. Providing such guarantees as "always link against libc and do so last" is ultimately for the Rustc authors to decide. Also Meson does always know that it is passing libraries written in C to rustc, so maybe it should always internally mark such libraries as dependent on libc?

Ultimately the proper place to fix this is for the rustc and/or meson authors to decide, I would just like to get Ubuntu moving forward.

[Mate Kukri (mkukri)]

I wonder why the Debian bug is suddenly marked fix released on here, I've tested meson 1.3.2-1 and it has the same problem.

[Mate Kukri (mkukri)]

Proposed fix (or maybe workaround?) for the library ordering issue.

If there are any C ABI libraries in the dependency list of a rustc linked target, with suffix them with `-Clink-args=-lc`.

Local testing shows good signs.

Currently being PPA built here: https://launchpad.net/~mkukri/+archive/ubuntu/dev/+build/27777096

autopkgtest against PPA https://autopkgtest.ubuntu.com/results/autopkgtest-noble-mkukri-dev/noble/arm64/m/meson/20240215_155251_95dff@/log.gz

[Oibaf (oibaf)]

> I wonder why the Debian bug is suddenly marked fix released on here

It's explained in the bug description:

"I have immediately closed this bug with the version in unstable, so if that version or a later version migrates, this bug will no longer affect testing."

[Oibaf (oibaf)]

> Proposed fix (or maybe workaround?) for the library ordering issue.

Great! Did you plan to submit it upstream?

[Mate Kukri (mkukri)]

@oibaf

Ah I understand that now, I just blanked and missed for a second this was always marked "Fix Released" for Debian on here.

I believe the DM for this package (jpakkane) is also one of the main upstream maintainers (and designer) of Meson, I wonder about his opinion on whether this is appropriate for inclusion in Debian and/or Meson? I am currently of the belief that rustc cannot internally know that a specific library given to it on a command line depends on libc so it needs to be told some way, but maybe it would be reasonable for rustc to always link with libc last.

However the above is a rather subtle point, and this patch does fix the library ordering issue at least for the specific case of implicit libc use, so I don't see it doing anymore harm in being Ubuntu. This however isn't my primary domain, so I'd rather leave a "proper" fix up-to someone who is more interested in these kinds of problems.

[Mate Kukri (mkukri)]

This was reported upstream at: https://github.com/mesonbuild/meson/issues/12877

[Jussi Pakkanen (jpakkane)]

> and it's rustc itself that emits `-lc` at the wrong location.

My reading of the issue is that Meson does not specify anything regarding `-lc`, but `rustc` adds it by itself. Is this correct?

If yes and Meson starts putting `-Clink-arg=-lc` on the command line, does `rustc` still add its own `-lc` or not?

Depending on the answers to the questions above, there may be two different bugs at play:

1. `rustc` puts its own `-lc` in the wrong place
2. Meson does not tell `rustc` that the libraries it is linking against need to be linked against the C runtime lib

If the bug is the former, then it should be fixed in `rustc`. If it is the latter then it should arguably be fixed in Meson. Even in this case it would be better for `rustc` to have a compiler argument like `--link-against-system-C-library` as opposed to a plain `-lc` because it is more declarative than imperative.

In any case it would be nice to get comments from `rustc` developers or maintainers. I don't know any and I don't think they have commented on this issue.

[Jussi Pakkanen (jpakkane)]

For reference, there is code in Meson that explicitly removes `-lc`.

https://github.com/mesonbuild/meson/blob/master/mesonbuild/compilers/rust.py#L111

[Mate Kukri (mkukri)]

@jpakkane

I think what is going on is that rustc adds `-lc` but at a different place because its own libraries depend on libc, but it doesn't know that for external libraries.

When specifying `-Clink-arg=-libfoo.a -Clink-arg=-lc`, it seems to know that libfoo that.a also depends on the libc.

My question is, should rustc know that automatically, or is specifying it like that reasonable?

[Simon Chopin (schopin)]

Disclaimer: despite my involvement on-and-off with the Rust toolchain in Ubuntu, I haven't nor plan to look at what rustc is doing here.

If we have a workaround for the issue, I'll take it and upload it to noble, for the simple reason that meson is blocking the python3.12 transition. We can very well revisit this and drop the patch later if it turns out to be rustc's fault (which is likely) and reasonably fixable there (still unknown), or even if we have a better way to do it.

I'm tagging this for the team to look at during our weekly meeting, to do a proper investigation on the Rust side.

[Launchpad Janitor (janitor)]

This bug was fixed in the package meson - 1.3.2-1ubuntu1

---------------
meson (1.3.2-1ubuntu1) noble; urgency=medium

  * d/p/lp2049904.patch: Suffix rustc link args with `-Clink-arg=-lc` when
    linking against external C ABI libraries (LP: #2049904)

 -- Mate Kukri <email address hidden> Thu, 15 Feb 2024 14:03:52 +0000

[Jussi Pakkanen (jpakkane)]

For reference, upstream rustc does have a flag for something along the lines of what was discussed above:

https://github.com/rust-lang/rust/pull/54675

[Mate Kukri (mkukri)]

@jpakkane I am not sure what's best for Meson to do, but at least to me it sounds reasonable for Meson to pass that flag when linking against C libraries.