[MIR] rustc, cargo, dh-cargo

Bug #1957932 reported by Simon Chopin
16
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cargo (Ubuntu)
Invalid
Critical
Simon Chopin
dh-cargo (Ubuntu)
Invalid
Undecided
Unassigned
rustc (Ubuntu)
Fix Released
Critical
Unassigned

Bug Description

[Availability]
The packages rustc and cargo are already in Ubuntu universe.
The packages build for the architectures they are designed to work on,
and are also built on platform with lesser upstream support, see
https://doc.rust-lang.org/nightly/rustc/platform-support.html for details.

They currently build and works for architectures:
  * amd64
  * arm64
  * armhf
  * i386
  * ppc64el
  * riscv64
  * s390x

Link to packages:

https://launchpad.net/ubuntu/+source/rustc
https://launchpad.net/ubuntu/+source/cargo
https://launchpad.net/ubuntu/+source/dh-cargo

Upcoming version:
https://launchpad.net/~schopin/+archive/ubuntu/rustc-mir/+sourcepub/13264343/+listing-archive-extra

[Rationale]
The packages rustc and cargo are required in Ubuntu main as the Rust programming
language is gaining in popularity, and those two packages are, respectively, its
main compiler implementation and its dedicated build tool (and dependency manager). dh-cargo is the standard packaging helper for Rust-based packages.

There are a few packages in main already that have partially switched to Rust
as their implementation language, and so rustc and cargo will be needed to keep
us in sync with their upstream. See for instance
https://bugs.launchpad.net/ubuntu/+source/mdevctl/+bug/1942394 and
https://lists.debian.org/debian-python/2021/12/msg00000.html
(python-cryptography is in main)

Note that the huge majority of our users will not use these packages, their
purpose is to be a build-dependency for other packages. In particular, it is
not particularly expected at this stage that those of our users that are Rust
developers, which usually rely on their toolchain being managed in their $HOME
by the `rustup` tool.

[Security]
cargo and rustc had 19 recorded security issues in the past, mostly in the Rust standard library
(1 affecting cargo):

https://nvd.nist.gov/vuln/search/results?form_type=Advanced&results_type=overview&search_type=all&isCpeNameSearch=false&cpe_vendor=cpe%3A%2F%3Arust-lang&cpe_product=cpe%3A%2F%3A%3Arust

All issues are usually handled promptly by the Rust team. However, the fixes
are rarely (if ever) backported to previous releases besides an occasional
1.X.1 point release for the latest stable.

There is an official Rust Security working group that curates a database of security
issues within the Rust ecosystem, including rustc/cargo:

https://github.com/rustsec/advisory-db

There are no history of known security issues with dh-cargo.

- no `suid` or `sgid` binaries
- no executables in `/sbin` and `/usr/sbin`
- Package does not install services, timers or recurring jobs
- Packages does not open privileged ports (ports < 1024)
- Packages does not contain extensions to security-sensitive software
(filters, scanners, plugins, UI skins, ...)

Note however that in typical use outside of packaging, building a project with cargo+rustc involves
executing code that has been downloaded from crates.io: cargo builds and executes
the `build.rs` file for any pre-compilation task (a bit like a Makefile), and any use
of a proc macro dependency basically implies running arbitrary code (the macro) within
the execution context of rustc.

[Quality assurance - function/usage]
The packages work well right after install, one can easily create a simple Rust project
and run it.

[Quality assurance - maintenance]
The packages do not deal with exotic hardware we cannot support

[Quality assurance - testing]
The cargo and rustc packages both run a test suite on build time. However, the rustc test suite
does NOT make the build fail as of 1.57. The reason is that there are always a few tests that fail, and it was a tradeoff made due to limited resources. Please note that Debian has a strategy of only failing the build if there are *too many* errors. As the Foundations team commits more resources on this toolchain, we've reverted back to Debian's system and are planning to making the testing story more rigorous.

Neither package has any autopkgtests in the versions currently in the release pocket. The upcoming rustc upload will have an autopkgtest consisting of rebuilding itself. Debian's cargo package now has a similar autopkgtest, that will be cherry-picked in the next cargo upload.

dh-cargo has neither build-time tests nor autopkgtests.

[Quality assurance - packaging]
debian/watch is present and works, dh-cargo is a native package.

rustc yields quite a bit of lintian output, but they seem mostly harmless.
https://lintian.debian.org/sources/rustc

There are lintian overrides present, but those are usually justified in the override
files themselves.

cargo is a bit more tame on the warning fronts
https://lintian.debian.org/sources/cargo

It has an override file for the source package, for relatively minor warnings.

dh-cargo is lintian-clean.

These packages does not rely on obsolete or about to be demoted packages.

The packages will not be installed by default

The packaging is fairly complex, especially in the case of rustc. The crux of
the complexity can be attributed to the boostrapping issue, as well as the need
to deal with vendored dependencies. There is extensive documentation within the
debian/ directories to help with the difficulties, and a lot of things have been
automated in scripts living in debian/scripts/*

Note that originally, the upstream tarball for rustc includes the sources for
cargo as well as its vendored dependencies, but the Debian Rust team chose to
split it out in its own package.

[UI standards]
I do not believe there's a need for translation for these applications given the
stated purpose for having them in main.

[Dependencies]
The packages have quite a few vendored dependencies. However, their non-vendored
dependencies are all in main, as well as the build-dependencies, assuming for the purpose
of this analysis that src:rustc is in main.

[Standards compliance]
The packages violate the Debian Policy on vendored
dependencies, with a copy of libgit2 (as a delta to Debian) in cargo.

The upcoming 1.58.1+dfsg1~ubuntu1 version of src:rustc unbundles the LLVM copy that's currently in src:rustc.

[Maintenance/Owner]
Owning Team will be Foundations
Team is not yet subscribed to all packages (dh-cargo still missing)

Both cargo and rustc packages use static linking for the Rust dependencies, as well as any
vendored C dependency (i.e. libgit2).

rustc uses vendored rust code tracked in Cargo.lock as shipped, in the package,
refreshing that code works via `cargo update ...` + cargo vendor. However, upgrading a single crate is more invasive than what could be expected, so patching vendored versions directly might be more advisable, as experienced with CVE-2022-23639.

cargo has a different process, documented in debian/README.source.
TODO: actually try to update a dependency in cargo using the debian/README.source,
which are not that clear.

[Background information]
The Package descriptions explains the package well
Upstream Name is developed by the Rust Compiler team and the Cargo team, under
the umbrella of the Rust Foundation
Link to upstream project: https://www.rust-lang.org/

Simon Chopin (schopin)
description: updated
Simon Chopin (schopin)
Changed in cargo (Ubuntu):
status: New → Incomplete
Changed in rustc (Ubuntu):
status: New → Incomplete
Lukas Märdian (slyon)
summary: - MIR: rustc, cargo
+ [MIR] rustc, cargo
Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote : Re: [MIR] rustc, cargo

Thanks for working on this Simon!

On the testing front, I think this is just pragmatism. There are always a few failing tests for not very interesting reasons, and a few more on !amd64. Debian does something slightly different and asserts no more than a certain (per-architecture) number of test cases fail. If we have the resources to look into these and patch out the tests we don't care about and try to file bugs upstream and generally stay on top of the situation, I think this would be a good thing. But the long turnaround in build times for rustc would make this a very tedious process. Maybe we can set up some CI or something.

Another point that's worth bearing in mind is that up to now the main point of maintaining the packages at all has been to deliver updates to firefox in LTS releases, so they have always been maintained with more of an eye to an easy stable update than being a good citizen in the devel series (this is what motivates the bundling of llvm, in particular). It would be reasonably easy to not vendor llvm in the devel series I think (although this would require us to be more aggressive about moving new versions of llvm to main I suspect).

I don't really understand why Debian separates the rustc and cargo source packages. I think we should consider not doing that, and in general doing things more the upstream way and not the Debian way (although not completely: I think upstream bundles the openssl source for example!).

A rustc autopkgtest that ensures rustc can build itself would be a good thing to have. We had a bug that prevented rustc from building itself on s390x get through to release once and re-bootstrapping to get out of this situation is painful.

Simon Chopin (schopin)
description: updated
Lukas Märdian (slyon)
Changed in rustc (Ubuntu):
milestone: none → ubuntu-22.04-beta
Changed in cargo (Ubuntu):
milestone: none → ubuntu-22.04-beta
Changed in rustc (Ubuntu):
importance: Undecided → Critical
Changed in cargo (Ubuntu):
importance: Undecided → Critical
Revision history for this message
Lukas Märdian (slyon) wrote :

Setting review priorities and milestones and moving status to "New" as I have been told this is now ready for review.

Changed in rustc (Ubuntu):
status: Incomplete → New
Changed in cargo (Ubuntu):
status: Incomplete → New
Changed in rustc (Ubuntu):
assignee: nobody → Christian Ehrhardt  (paelzer)
Changed in cargo (Ubuntu):
assignee: nobody → Didier Roche (didrocks)
Simon Chopin (schopin)
description: updated
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :
Download full text (11.8 KiB)

Review for Package: rustc

[Summary]
MIR team ACK under the constraint to resolve the below listed
required TODOs and as much as possible having a look at the
recommended TODOs.

This does need a security review, so I'll assign ubuntu-security
In addition security has to check and state if keeping the embeeded
llvm is ok for them.

List of specific binary packages to be promoted to main:
  rustc, libstd-rust-dev, libstd-rust-1.57
Specific binary packages built, but NOT to be promoted to main:
  rust-doc, rust-src, rust-all
Specific binary packages built yet unsure if they will be promoted,
@Foundations please clarify if you plan to seed any of those in main as well:
  rust-gdb, rust-lldb, rust-clippy, rustfmt

This qas quite a complex case (as expected), I hope I didn't miss too much.
In any case if there are comments by others in regard to the case that you think
need to be done for full support that I missed to mention feel free to speak up
please and add a comment.

This case has rather many notes, required and recommended todos.
For a better reference to them as you are going to anser/implement them
I'm numbering them. Please when fixin/answering them refer to them by that
number so that we can more easily check later if all got adressed.

## Notes:

#1 - libstd-rust is known to be very light and almost whatever you do you need
  further libs. We have in the context of this done an evaluation and identified
  some very common libs (log, serde, tempfile, structopt -open for suggstions)
  of almost always used libraries which I'd highly recommend to eventually MIR
  as part of the base rust support to make sense overall and to provide support
  for commonly used bits.
  OTOH the current draft [1] of MIR rules for other rust using packages suggests
  to vendor everything - until that changes the current situation is ok. But
  when this ever changes the general rust support will need a set of the most
  common libraries in main as the stdlib never is enough.

#2 - llvm-13 is needed in main which should soon happen anyway, not an extra task
  here. The one potential problem there is that so far never lldb-$ver was in
  main but with rustc it will be needed. I didn't see an issue as the source is
  in main and the further deps as well, but I have no deep insight into llvm,
  there might be issues if one goes down that path in more detail.

## Required TODOs:

#3 - As part of the rules draft for later MIR of rust based packages [1] a few
  structural requirements were identified which aren't part of rustc but the
  rustc ecosystem. In particular that would include the following two tasks
  that would need to be compleded to support rustc in a useful way for the
  distribution overall. Those likely need to happen in dh-cargo:
  #3a - the proper creation of a post-build .lock file and providing it in a
    standard path like /usr/share/doc/$pkg/
  #3b - Current build environments create X-Cargo-Built-Using but for plenty of
    Ubuntu mechanisms it would be required to put that data in Built-Using
    e.g. Component mismatch checks
  #3c - Even rustc itself has 288 vendored rust sources, so we will need both of
    the above .lock and built-usin...

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Back to incomplete until at least the open questions e.g. if you want to go with embedded or system llvm are answered.

Once you have answered those please set rustc back to "New" and assign "ubuntu-security".
It can be in their queue for reviews at the same time that you are then working on the remaining required and recommended tasks.

Changed in rustc (Ubuntu):
assignee: Christian Ehrhardt  (paelzer) → Simon Chopin (schopin)
status: New → Incomplete
Revision history for this message
Simon Chopin (schopin) wrote :

Hi,

This is just a partial update while I'm working on the rustc packaging. We're still debating the LLVM situation (item #10), but once that's done I'll upload the newer 1.58.1 which comes with the security fix for CVE-2022-21658 (item #13). I'm also looking into the test suite issue (items #5, #11)

Regarding #3a the new package will ship with the Cargo.lock file in /usr/share/doc/rustc (I've confirmed that it is regenerated at build time so the dependencies are the correct ones).
I don't really understand the point of #3b and #3c in the context of src:rustc, as packaged dependencies aren't statically linked, and the rustc packaging isn't using dh-cargo, which is the component that generates the mentioned fields in the first place.

For #7 and #8, I wasn't really aware that I could exclude binaries! For src:rustc, I'd only ship in main rustc, rustdoc, and their dependencies. As mentioned in the MIR, I don't expect Rust developers to actually use the packaged tools at this point.

Items with no action so far: #4, #6, #9, #12
I'll update the bug description once the LLVM issue is resolved.

Revision history for this message
Didier Roche-Tolomelli (didrocks) wrote :
Download full text (5.7 KiB)

Review for package: cargo

List of specific binary packages to be promoted to main was proposed only cargo, but cargo-doc could go as well IMHO.

MIR team ACK given the constaint of having the security-review processed as it downloads crate from the Internet and acked and the Required TODOs processed
Notes:
Required TODOs:
- There is no autopkgtests and the bug description has a "TODO: cargo autopkgtest that creates a new crate, adds a simple dependency
(anyhow is a good candidate, no transitive deps), vendors it, builds the binary from the vendored crate, and runs it."
I suggest either proceeding with this plan OR write a manual tests before release. I think the autopkgtest has a benefit impact as it will validate rdeps.
- debian/copyright is not correct. For instance vendor/libgit2-sys/ is listed as Apache2/MIT, but it’s distributed under the GNU GPL v2 with a Linking Exception. The linking exception does not mention Apache2/MIT explicitly but give you unlimited permission to link the compiled version of this library into combinations with other programs, and to distribute those combinations without any restriction coming from the use of this file. I do not think Apache2/MIT is completely accurate and debian/copyright should reflect GPL2 + Linking exception.
A lot of other files are in a similar situation and I suggest you to refresh the debian/copyright file. Maybe I’m wrong and this is a valid "relicensing" in that case, but I would prefer that to be stated explicitly.

Recommended TODOs:
- Ubuntu does carry a delta since November 2020, which could be large. It would be good to either resync or decide that we diverged from it forever (and so, considering remerging rustc and cargo together)
- One lintian error is about a malformed override. It would be good looking into this.
- the vendor directory has 2 files which were due to patch failure:
W: cargo source: debian-adds-patch-failure-file vendor/libnghttp2-sys/build.rs.orig
W: cargo source: debian-adds-patch-failure-file vendor/openssl-sys/Cargo.toml.rej
Would be good to fix those.
- Have a look if the numerous warnings are reported usptream and if they are worked on.

[Duplication]
There is no other package in main providing the same functionality.

[Dependencies]
OK:
- no other Dependencies to MIR due to this outside rustc, which is part of the MIR
   - checked with check-mir
   - not listed in seeded-in-ubuntu
   - none of the (potentially auto-generated) dependencies (Depends
     and Recommends) that are present after build are not in main
  - no -dev/-debug/-doc packages that need exclusion
  - No dependencies in main that are only superficially tested requiring more tests now.

[Embedded sources and static linking]
OK:
- no embedded source present
- some static linking, but it’s rustc as part of rust synced source.
- does not have odd Built-Using entries

OK:
- not a go package, no extra constraints to consider in that regard
- vendoring is used, but the reasoning is obvious due to rustc being split as a separate package by debian.

[Security]
OK:
- history of CVEs does not look concerning
- does not run a daemon as root
- does not use webkit1,2
- does not use lib*v8 directly
- d...

Read more...

Changed in cargo (Ubuntu):
assignee: Didier Roche (didrocks) → Simon Chopin (schopin)
status: New → Incomplete
Revision history for this message
Simon Chopin (schopin) wrote :
Download full text (3.6 KiB)

Status of the rustc part of the MIR:

There's a package available at https://launchpad.net/~schopin/+archive/ubuntu/rustc-mir/+packages (still building...). I'm aware that there are issues with the armhf build and am still investigating them. Barring that, I feel the package is ready for security review.

I'll walk through the items from cpaelzer review.

I'll leave #1 for a later date (and MIR :D), but I do have the issue in mind.

#2 has been resolved by reverting the bundling of LLVM. We expect to be more proactive in the Foundations team regarding rustc and LLVM, and the latter needs regular upgrades for other components of our releases as well (kernel and mesa)

#3: I've shipped Cargo.lock in /usr/share/doc/rustc/Cargo.lock, the exact location can be revisited. I've checked, it is indeed generated at build time and so is up to date. Note that src:rustc does *not* use dh-cargo so the related points aren't applicable.

#4: Still no action on MIRing dh-cargo at this time.

#5: I've added a self-build autopkgtest to rustc. It actually already caught a small regression in their linkchecker tool that impacts us because we de-bundle the JS and fonts from the doc.

#6: The testsuite is run on s390x, it "just" has a higher failure tolerance than other architectures, with 40 failures allowed (8 for amd64)

#7, #8: I still don't know how to do the exclude definition :-/. As stated earlier, I think the scope of the MIR should be for the rustc and libstd-rust-* packages, with the -doc and all ancillary tooling remaining in universe as they're not essential to the packaging of Rust applications.

#9: LTO has been explicitly disabled in d/rules, with a comment to actually revisit this decision to see if it's still needed.

#10: We'll go with separate LLVM, I'll update the bug description.

#11: Tests have been re-enabled using Debian's "not too many errors" rules as a first step, and I plan on cataloging failures to narrow the policy, but it won't be possible in the near future.

#12: foundations-bugs is now subscribed to both src:rustc and src:cargo

#13: I worked on 1.58.1 directly, as I needed to do some repacking anyway to remove some vendored dependencies that didn't need to be in there (e.g. the curl and openssl crates)

Regarding the MANY warnings during the build, those warnings are basically linting for constructs that used to be necessary but aren't anymore as the compiler gets smarter, and, at least when compiling with the "right" rustc version (meaning the N-1 version), are all explicitly allowed. However, since we need to be able to compile with the *current* version, we've had to lift the blanket deny instruction for all *other* warnings, as new warnings introduced by the version compiled won't be in this allowed list, as we've experienced when working on 1.57. Going forward, an axis of improvement would be to see if it would be possible to actually suppress all the benign warnings to make the new ones stand out.

Regarding armhf:

When mentioning on IRC that the armhf build had some issue, a question was raised of whether we should enable rustc on this architecture at all, especially since rustc tends towards heavy memory consumption...

Read more...

Simon Chopin (schopin)
description: updated
Simon Chopin (schopin)
Changed in rustc (Ubuntu):
assignee: Simon Chopin (schopin) → Ubuntu Security Team (ubuntu-security)
Revision history for this message
Simon Chopin (schopin) wrote :

Rustc update: I plan on uploading rustc 1.58.1 to the archive tomorrow before feature freeze, as I've identified the failing tests as being failing ever since Impish, and not "mission-critical", as they are related to debug info, which is IMO mostly for developers, making it out of scope for this MIR :)

Revision history for this message
Steve Beattie (sbeattie) wrote :

I reviewed rustc 1.57.0+dfsg1+llvm-0ubuntu2 as checked into jammy
(but also peeked briefly at 1.58.1+dfsg1~ubuntu1-0ubuntu1~ppa5
in Simon's ppa). This shouldn't be considered a full audit but
rather a quick gauge of maintainability, and this is a bit more
streamlined review than normal due to the nature of Rust.

Rust is a programming language and runtime environment that is
intended to be a modern systems language. In general, the Ubuntu
Security team views more widespread usage of Rust as a positive
thing; the primary drawback being, like Go before it, the choice
to static link everything makes security updates more challenging
for both the deliverer and users on limited bandwidth.

The Built-Using: mechanism at least gives us a chance to determine
what needs to be rebuilt when a rust library has a security
vulnerability that needs addressing. In order to get Built-Using:
applied to Rust applications in jammy, does this mean that every
Rust application needs at a minimum a no-change rebuild before
jammy is released? If so, is there a plan for that?

I'd like to ask what is the support expectation and commitment
from the Foundations team for the rust toolchain and the separated
out LLVM:

 - Is the expectation that version bumps of rust, possibly along
   with version bumps of LLVM necessary, will be brought back to
   22.04 LTS?

 - If so, does the source package need a versioned name, as done
   for other toolchains?

 - As more thing depend on rust either wholly or partially
   (e.g. the ongoing work on the Linux kernel), is there an
   expectation this will change for 24.04 LTS?

For CVE history, there are 21 CVEs in the security team's tracker
that affect Rust, 20 in the standard library. (There is also a
very recent additional issue that affects the vendored copy of
rust-crossbeam in the rustc source package.) Generally, upstream
looks responsive to security issues.

Given all the above, the Ubuntu Security provisionally acks rustc
for main, assuming the questions above can be answered.

Changed in rustc (Ubuntu):
assignee: Ubuntu Security Team (ubuntu-security) → nobody
Revision history for this message
Simon Chopin (schopin) wrote :

Built-Using is already used in all pure-Rust binaries, e.g. ripgrep. However, it only stores the version of rustc itself. The non-vendored libraries are currently recorded using the non-standard field 'X-Cargo-Built-Using'. Sadly, this doesn't apply to all packages that currently Build-Depend on rustc, only those that use the dh-cargo helper. src:0ad for instance doesn't seem to record anything for that, nor does src:cargo, ironically.

Furthermore, in the case of vendored dependencies, we've settled for storing the "Cargo.lock" file which is a manifest of all the dependencies of a given project with precise versioning. Currently for rustc we ship it in /usr/share/doc/rustc/Cargo.lock

Regarding version bumps, it is worth noting that they already happen in our other LTS versions. rustc 1.57 is indeed available in focal-updates and bionic-updates. AIUI it's because of Firefox regularly needing a newer version of rustc. Similarly, newer versions of LLVM are available on older releases because of kernel and mesa requirements.

However, the versioned source name is a good point. On one hand it would make sense because even though the rustc upstream tries their best not to break existing code, breakage does occur from time to time, so versioned rustc would make transitions easier. OTOH, rustc requires the N-1 version to build (or itself), which would mean that we would have the whole integer suite in the archive, which doesn't seem particularly appealing to me given their 6-week cadence. I'll put it on Foundation's agenda for the sprint next week.

I honestly cannot give you any expectations of how our handling of the toolchain for 24.04 would be, as it will be shaped by the requirements of the projects that will use it. I personally have some doubts that we'll see *production* Rust code in our kernels for 24.04. I also suspect that as the ecosystem matures we'll see less and less breaking changes in rustc, as the long trail of unmaintained crates in crates.io will calcify things a bit (they compile and run the tests for each crate there for each release).

Thanks for the heads-up on the crossbeam issue, I'll update the affected vendored version before uploading.

Revision history for this message
Matthieu Clemenceau (mclemenceau) wrote :

@Steve Beattie,

I concur with Simon, the foundations team will backport llvm to 22.04 if an updated rustc requires it.

Matthieu

Revision history for this message
Simon Chopin (schopin) wrote :

After sitting down with some members of the MIR team and an archive admin, here's our plan for future rustc releases:

We'll go for versioned source packages, so that packages that are stuck depending on older rustc versions for some reason can still work. To avoid having too many versions entering the main archive though, we'll upload all the releases to a PPA and will binary-copy the versions that are required by one of the packages, e.g. Firefox. Once that version hits the archive, we 'll upload a no-change version to ensure that the package in the archive is reproducible. Essentially, we're doing a sort of bootstrap for each version.

I still need to figure out the details of how to implement this without breaking existing code, that'll likely be tomorrow's update.

Revision history for this message
Simon Chopin (schopin) wrote :
Simon Chopin (schopin)
Changed in rustc (Ubuntu):
status: Incomplete → Confirmed
status: Confirmed → In Progress
Revision history for this message
Simon Chopin (schopin) wrote :

@paelzer I'm not exactly sure of where we are in the status diagram for rustc in the MIR. I put it as "In Progress" by default.

Given the conclusion in bug #1964098 as well as the various answers brought in comments #7 and #10 I think I've addressed all comments. As I believe rustc to be useful in the archive even in the absence of cargo, as some build systems actually use it directly (e.g. meson), I've posted a MP for the seed changes. I'll toggle it as up for reviews if you agree with this: https://code.launchpad.net/~schopin/ubuntu-seeds/+git/ubuntu-seeds/+merge/416688

Revision history for this message
Steve Beattie (sbeattie) wrote :

> 'Built-Using' vs 'X-Cargo-Built-Using' dh-cargo behavior

So there is no plan to change this in dh-cargo? The tool the security team has that queries Built-Using can be modified to use the alternate field, if necessary, but we need to know if that's what we need to do. Are the tools that help with library transitions in Ubuntu going to cope with this?

> non-users of dh-cargo not emitting 'X-Cargo-Built-Using'

Is there a plan to deal with this? Some sort of britney / autopkgtest check that could be added to flag these as needing to be addressed? Otherewise, this makes it more difficult to discern what might need to be rebuilt given an update to a given rust library.

I do appreciate the Cargo.lock packaging, that is helpful, though it means neediing to unpack binary debs to gain access to them, rather than merely accessing archive metadata for 'Built-Using' or 'X-Cargo-Built-Using'.

Thanks.

Revision history for this message
Simon Chopin (schopin) wrote :
Download full text (6.4 KiB)

@sbeattie there's some context on those various fields in https://github.com/cpaelzer/ubuntu-mir/pull/3

Basically X-Cargo-Built-Using should be folded into Built-Using. There has been no talk of automating detection of packages that ought to have those fields, but that does sound like a good idea.

However, in the case of rustc and any future main package built using Rust, there are going to be vendored dependencies that are not packaged at all. It doesn't seem like a good idea to me to document those in the same fields as the dependencies that are separately packaged but statically linked, which is why I proposed shipping the Cargo.lock file.

If you'd prefer, we could instead ship it in another field, maybe X-Vendored-Sources (as mentioned before, Built-Using seems out of scope for that). For instance, using this small Python snippet, I get this for the Cargo.lock file shipped in rustc (Jammy):

$ zcat Cargo.lock.gz | python3 -c "import toml; import sys; print(', '.join(f\"{p['name']}/{p['version']}\" for p in toml.load(sys.stdin)['package'] if 'source' in p))"

addr2line/0.16.0, adler/1.0.2, aho-corasick/0.7.18, ammonia/3.1.0, annotate-snippets/0.8.0, ansi_term/0.11.0, ansi_term/0.12.1, anyhow/1.0.45, array_tool/1.0.3, arrayvec/0.7.2, atty/0.2.14, autocfg/1.0.1, bitflags/1.3.2, block-buffer/0.7.3, block-buffer/0.9.0, block-padding/0.1.5, bstr/0.2.13, byte-tools/0.3.1, bytecount/0.6.2, byteorder/1.3.4, camino/1.0.5, cargo-platform/0.1.2, cargo_metadata/0.12.0, cargo_metadata/0.14.1, cc/1.0.71, cfg-if/0.1.10, cfg-if/1.0.0, chalk-derive/0.55.0, chalk-engine/0.55.0, chalk-ir/0.55.0, chalk-solve/0.55.0, chrono/0.4.19, clap/2.33.3, cmake/0.1.44, colored/2.0.0, compiler_builtins/0.1.53, compiletest_rs/0.7.1, cpuid-bool/0.1.2, crc32fast/1.2.1, crossbeam-channel/0.5.1, crossbeam-deque/0.7.4, crossbeam-deque/0.8.1, crossbeam-epoch/0.8.2, crossbeam-epoch/0.9.5, crossbeam-queue/0.2.3, crossbeam-utils/0.7.2, crossbeam-utils/0.8.5, cstr/0.2.8, ctor/0.1.15, datafrog/2.0.1, derive-new/0.5.8, diff/0.1.12, difference/2.0.0, digest/0.8.1, digest/0.9.0, dirs/2.0.2, dirs-next/2.0.0, dirs-sys/0.3.6, dirs-sys-next/0.1.2, dlmalloc/0.2.3, either/1.6.1, elasticlunr-rs/2.3.9, ena/0.14.0, env_logger/0.7.1, env_logger/0.8.4, expect-test/1.0.1, fake-simd/0.1.2, filetime/0.2.15, fixedbitset/0.2.0, flate2/1.0.22, fnv/1.0.7, form_urlencoded/1.0.1, fortanix-sgx-abi/0.3.3, fs-err/2.5.0, futf/0.1.4, generic-array/0.12.4, generic-array/0.14.4, getopts/0.2.21, getrandom/0.1.14, getrandom/0.2.0, gimli/0.25.0, glob/0.3.0, globset/0.4.5, globwalk/0.8.1, gsgdt/0.1.2, handlebars/4.1.0, hashbrown/0.11.2, heck/0.3.3, hermit-abi/0.1.19, hex/0.4.2, html5ever/0.25.1, humantime/1.3.0, humantime/2.0.1, idna/0.2.3, if_chain/1.0.0, ignore/0.4.17, indexmap/1.7.0, indoc/1.0.3, instant/0.1.12, itertools/0.9.0, itertools/0.10.1, itoa/0.4.8, jobserver/0.1.24, jsonpath_lib/0.2.6, lazy_static/1.4.0, libc/0.2.107, libm/0.1.4, lock_api/0.4.5, log/0.4.14, lzma-sys/0.1.16, mac/0.1.1, macro-utils/0.1.3, maplit/1.0.2, markup5ever/0.10.0, markup5ever_rcdom/0.1.0, matchers/0.0.1, matches/0.1.9, maybe-uninit/2.0.0, md-5/0.9.1, mdbook/0.4.12, measureme/10.0.0, memchr/2.4.1, memmap2/0.2.1, memoffset/0.5.5, ...

Read more...

Revision history for this message
Steve Beattie (sbeattie) wrote : Re: [Bug 1957932] Re: [MIR] rustc, cargo

On Fri, Mar 11, 2022 at 10:17:47AM -0000, Simon Chopin wrote:
> @sbeattie there's some context on those various fields in
> https://github.com/cpaelzer/ubuntu-mir/pull/3

Thanks for this.

> Basically X-Cargo-Built-Using should be folded into Built-Using.

I agree with this, but is there a plan to land this in jammy?
If not, our tooling needs to compensate.

> There has been no talk of automating detection of packages that ought
> to have those fields, but that does sound like a good idea.

I think something needs to be in place, or there runs the risk of things
needing to pick up updates that don't get them applied.

> However, in the case of rustc and any future main package built using
> Rust, there are going to be vendored dependencies that are not packaged
> at all. It doesn't seem like a good idea to me to document those in the
> same fields as the dependencies that are separately packaged but
> statically linked, which is why I proposed shipping the Cargo.lock file.
>
> If you'd prefer, we could instead ship it in another field, maybe
> X-Vendored-Sources (as mentioned before, Built-Using seems out of scope
> for that).

It would be great if we could get this information as a field in the
Packages info (modulo concerns about size explosion as the set of
packaged rust software expands). I agree that it is not appropriate
for Built-Using; X-Vendored-Sources sounds great (if only we could
get this incorporated across more language ecosystems!). It would
probably be beneficial to have both the field in the packages metadata
list as well as the Cargo.lock file, to be able to identify which
crate versions were incorporated in superseded versions, if need be.

> For instance, using this small Python snippet, I get this for
> the Cargo.lock file shipped in rustc (Jammy):
>
> $ zcat Cargo.lock.gz | python3 -c "import toml; import sys; print(',
> '.join(f\"{p['name']}/{p['version']}\" for p in
> toml.load(sys.stdin)['package'] if 'source' in p))"

Thanks for this, lots to chew on here. Quite a few rust crates have
at least two versions of themselves in the list, which based on reading,
seems to be normal in the ecosystem, but then leads to issues like:

  crossbeam-utils/0.7.2
  crossbeam-utils/0.8.5

while the latter was patched to address CVE-2022-23639 in the current
jammy packaging, the former (in vendor/crossbeam-utils-0.7.2) was not.
While upstream crossbeam-utils yanked all of the 0.8.x versions < 0.8.7,
but there doesn't appear to be a fixed version of 0.7.x from upstream.
That's somewhat concerning about the ecosystem as a whole.

> The 'if source in p' statement filters out crates that are internal to
> rustc. Surprinsingly, the remaining rustc-* crates are separately
> packaged forks of existing crates.

That is also less than ideal.

> Would the security team feel more comfortable with this?

Yes, I think so.

Thanks!

Revision history for this message
Simon Chopin (schopin) wrote :

Before even starting to address the various points further, I must ask
whether they're showstopper for the *rustc* MIR.

I ask because some of the concerns raised here are irrelevant for rustc
itself. For instance, the X-Cargo-Built-Using is not only not used by
the rustc packaging at all, it would also not be used by Rust packages
entering main since, under the proposed amended rules, those packages
would first vendor all their dependencies.

We intend to implement all tooling changes that are required for a
wider Rust ecosystem support in main, but this starts with having the
compiler!

Revision history for this message
Steve Beattie (sbeattie) wrote :

On Tue, Mar 15, 2022 at 05:14:00PM -0000, Simon Chopin wrote:
> Before even starting to address the various points further, I must ask
> whether they're showstopper for the *rustc* MIR.

> I ask because some of the concerns raised here are irrelevant for rustc
> itself. For instance, the X-Cargo-Built-Using is not only not used by
> the rustc packaging at all, it would also not be used by Rust packages
> entering main since, under the proposed amended rules, those packages
> would first vendor all their dependencies.

Ah, I had missed this piece from the conversation on the github
MR. That places more emphasis on making sure Cargo.lock at a minimum
is included. Long term, it would be ideal to have these in package
metadata as X-Embedded-Copies or whatever, but ultimately that's
a feature that would be generally useful across the distro and in
Debian, not just in the Rust portions of it.

For X-Cargo-Built-Using vs Built-Using in dh-cargo, the Security team
can compensate one way or the other, we just need to know that, no,
Built-Using not going to land in jammy. With the intent to fully vendor
things in main, it's less important (from our team's perspective)
that this gets resolved one way or the other, but I note that we are
not the only ones with an opinion here.

One other consideration is that organizations and governments are
pushing really strongly for Software Bills of Materials (SBOMs) so
the more proactive we are about collecting needed information in a
structured, easily consumable way, the more straightforward it will
be to satisfy those requirements.

> We intend to implement all tooling changes that are required for a
> wider Rust ecosystem support in main, but this starts with having the
> compiler!

The reason I ask about ecosystem supportability here is because this
is likely the sole point where it's even in bounds for an MIR security
audit. The 'dh-cargo' package as a "build-time only" tool means there
is no requirement for it to go into main, and thus will likely never
receive an MIR. When it comes time to review cargo, the argument will
then be "Of what use is having rustc in main without cargo?" Individual
applications or libraries will have reviews focused on themselves.

The reality is we accepted Go-lang into main with a hypothetical plan
to support its ecosystem security-wise, but has been difficult to
turn into something real. My concern is that we're about to do the
same for Rust, despite our broad general approval of the language.

[There are also other constraints within Canonical that cause me
to be thinking about the supportability of the ecosystem as a whole
beyond what gets integrated into main, but you are correct that they
are out of bounds for an MIR.]

Anyhow. I have concerns about supporting this ecosystem, but the
provisional ACK is already there.

Thanks.

Revision history for this message
Simon Chopin (schopin) wrote : Re: [MIR] rustc, cargo

I think the XS-Vendored-Copy or whatever should be split out onto a ML thread so that we can work on something that's equally applicable for the Go ecosystem, even if retroactively. I'll take care of it soon.

I'm preparing an upload for rustc that fixes the crossbeam-utils CVE in *both* copies, this time. However, I'd rather not do multiple uploads, so I need a clarification from paelzer:

@paelzer, assuming rustc gets to main, do we need to downgrade the Recommends: cargo into a Suggests?

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

> @paelzer, assuming rustc gets to main, do we need to downgrade the Recommends:
> cargo into a Suggests?

Yes if Cargo isn't ready for promotion to main yet you can't promote anything that has a Recommends/Depends onto it.

Revision history for this message
Simon Chopin (schopin) wrote :

Regarding the Suggests downgrade, I filed an FFe before uploading:
https://bugs.launchpad.net/ubuntu/+source/rustc/+bug/1966200

Regarding the Vendored-Copy field, I opened a discussion on the MIR rules changes directly as I'd rather have the thing hashed out before changing the rustc packaging. Be assured though that I will follow up on it for rustc: https://github.com/cpaelzer/ubuntu-mir/pull/3/files#r833109362

Simon Chopin (schopin)
summary: - [MIR] rustc, cargo
+ [MIR] rustc, cargo, dh-cargo
Simon Chopin (schopin)
description: updated
Simon Chopin (schopin)
Changed in dh-cargo (Ubuntu):
status: New → Incomplete
Revision history for this message
Simon Chopin (schopin) wrote (last edit ):

Now that the new rustc has migrated from -proposed, I'd like to move forward with the rustc MIR, as I believe all the issues raised during its review (#3) have been addressed one way or the other, see #7 and subsequent updates since.

We also have a provisional ACK from the security team (I'll keep working on surfacing the vendored deps data in a better way than Cargo.lock!).

The seed changes are in a MP at https://code.launchpad.net/~schopin/ubuntu-seeds/+git/ubuntu-seeds/+merge/416688

@paelzer could you confirm that we can move ahead, and perhaps review the seed change?

Revision history for this message
Steve Beattie (sbeattie) wrote : Re: [Bug 1957932] Re: [MIR] rustc, cargo, dh-cargo

On Mon, Apr 04, 2022 at 09:31:39AM -0000, Simon Chopin wrote:
> We also have a provisional ACK from the security team (I'll keep working
> on surfacing the vendored deps data in a better way than Cargo.lock!).
>
> The seed changes are in a MP at
> https://code.launchpad.net/~schopin/ubuntu-seeds/+git/ubuntu-
> seeds/+merge/416688
>
> @paelzer could you confirm that we can move ahead, and perhaps review
> the seed change?

From the Ubuntu Security Team's perspective, ACK for moving ahead.

Thanks!

Revision history for this message
Simon Chopin (schopin) wrote :

The seed has been updated, we now need an AA to promote the following binaries: libstd-rust-1.58 libstd-rust-dev rustc

Changed in rustc (Ubuntu):
status: In Progress → Fix Committed
Revision history for this message
Steve Langasek (vorlon) wrote :
Download full text (6.5 KiB)

Override component to main
rustc 1.58.1+dfsg1~ubuntu1-0ubuntu2 in jammy: universe/misc -> main
libstd-rust-1.58 1.58.1+dfsg1~ubuntu1-0ubuntu2 in jammy amd64: universe/libs/optional/100% -> main
libstd-rust-1.58 1.58.1+dfsg1~ubuntu1-0ubuntu2 in jammy arm64: universe/libs/optional/100% -> main
libstd-rust-1.58 1.58.1+dfsg1~ubuntu1-0ubuntu2 in jammy armhf: universe/libs/optional/100% -> main
libstd-rust-1.58 1.58.1+dfsg1~ubuntu1-0ubuntu2 in jammy i386: universe/libs/optional/100% -> main
libstd-rust-1.58 1.58.1+dfsg1~ubuntu1-0ubuntu2 in jammy ppc64el: universe/libs/optional/100% -> main
libstd-rust-1.58 1.58.1+dfsg1~ubuntu1-0ubuntu2 in jammy riscv64: universe/libs/optional/100% -> main
libstd-rust-1.58 1.58.1+dfsg1~ubuntu1-0ubuntu2 in jammy s390x: universe/libs/optional/100% -> main
libstd-rust-dev 1.58.1+dfsg1~ubuntu1-0ubuntu2 in jammy amd64: universe/libdevel/extra/100% -> main
libstd-rust-dev 1.58.1+dfsg1~ubuntu1-0ubuntu2 in jammy arm64: universe/libdevel/extra/100% -> main
libstd-rust-dev 1.58.1+dfsg1~ubuntu1-0ubuntu2 in jammy armhf: universe/libdevel/extra/100% -> main
libstd-rust-dev 1.58.1+dfsg1~ubuntu1-0ubuntu2 in jammy i386: universe/libdevel/extra/100% -> main
libstd-rust-dev 1.58.1+dfsg1~ubuntu1-0ubuntu2 in jammy ppc64el: universe/libdevel/extra/100% -> main
libstd-rust-dev 1.58.1+dfsg1~ubuntu1-0ubuntu2 in jammy riscv64: universe/libdevel/extra/100% -> main
libstd-rust-dev 1.58.1+dfsg1~ubuntu1-0ubuntu2 in jammy s390x: universe/libdevel/extra/100% -> main
rust-all 1.58.1+dfsg1~ubuntu1-0ubuntu2 in jammy amd64: universe/devel/optional/100% -> main
rust-all 1.58.1+dfsg1~ubuntu1-0ubuntu2 in jammy arm64: universe/devel/optional/100% -> main
rust-all 1.58.1+dfsg1~ubuntu1-0ubuntu2 in jammy armhf: universe/devel/optional/100% -> main
rust-all 1.58.1+dfsg1~ubuntu1-0ubuntu2 in jammy i386: universe/devel/optional/100% -> main
rust-all 1.58.1+dfsg1~ubuntu1-0ubuntu2 in jammy ppc64el: universe/devel/optional/100% -> main
rust-all 1.58.1+dfsg1~ubuntu1-0ubuntu2 in jammy riscv64: universe/devel/optional/100% -> main
rust-all 1.58.1+dfsg1~ubuntu1-0ubuntu2 in jammy s390x: universe/devel/optional/100% -> main
rust-clippy 1.58.1+dfsg1~ubuntu1-0ubuntu2 in jammy amd64: universe/devel/optional/100% -> main
rust-clippy 1.58.1+dfsg1~ubuntu1-0ubuntu2 in jammy arm64: universe/devel/optional/100% -> main
rust-clippy 1.58.1+dfsg1~ubuntu1-0ubuntu2 in jammy armhf: universe/devel/optional/100% -> main
rust-clippy 1.58.1+dfsg1~ubuntu1-0ubuntu2 in jammy i386: universe/devel/optional/100% -> main
rust-clippy 1.58.1+dfsg1~ubuntu1-0ubuntu2 in jammy ppc64el: universe/devel/optional/100% -> main
rust-clippy 1.58.1+dfsg1~ubuntu1-0ubuntu2 in jammy riscv64: universe/devel/optional/100% -> main
rust-clippy 1.58.1+dfsg1~ubuntu1-0ubuntu2 in jammy s390x: universe/devel/optional/100% -> main
rust-doc 1.58.1+dfsg1~ubuntu1-0ubuntu2 in jammy amd64: universe/doc/extra/100% -> main
rust-doc 1.58.1+dfsg1~ubuntu1-0ubuntu2 in jammy arm64: universe/doc/extra/100% -> main
rust-doc 1.58.1+dfsg1~ubuntu1-0ubuntu2 in jammy armhf: universe/doc/extra/100% -> main
rust-doc 1.58.1+dfsg1~ubuntu1-0ubuntu2 in jammy i386: universe/doc/extra/100% -> main
rust-doc 1.58.1+d...

Read more...

Changed in rustc (Ubuntu):
status: Fix Committed → Fix Released
Revision history for this message
Steve Langasek (vorlon) wrote :

(binaries will be re-demoted as necessary)

Revision history for this message
Steve Langasek (vorlon) wrote :

rustc has been promoted without a need to promote cargo; and the tasks on the other packages are marked incomplete (maybe they should be closed?). Nothing further here for ubuntu-archive to do at the moment, so unsubscribing.

Revision history for this message
Simon Chopin (schopin) wrote :

Closing the dh-cargo and cargo tasks as Invalid in favor of the new bug, for the sake of clarity:

https://bugs.launchpad.net/ubuntu/+source/cargo/+bug/1993819

Changed in cargo (Ubuntu):
status: Incomplete → Invalid
Changed in dh-cargo (Ubuntu):
status: Incomplete → Invalid
Revision history for this message
Lukas Märdian (slyon) wrote :

In order to untangle this process (see cargo MIR, bug #1993819), we decided to demote src:rust to universe, until all of rustc and cargo is clear from MIR and security POV. Then, the new (combined) package can be re-promoted into "main".

In the meantime, Zixing can continue working on rustc in universe to satisfy the Firefox and Kernel needs.

Revision history for this message
Lukas Märdian (slyon) wrote :

This seems ready for re-promotion, now. After bug #1993819 bug #1991650 and bug #1990655 got resolved.

Changed in rustc (Ubuntu):
status: Fix Released → New
Revision history for this message
Mark Esler (eslerm) wrote :

Thank you @slyon.

Security's ask for #1991650 did not seem reasonable, so I have changed the conditional ACK.

I believe Security has now provided a path for promotion of rustc and cargo in Mantic.

@liushuyu-011, please let me know if you need anything.

Revision history for this message
Mark Esler (eslerm) wrote :

rustc (containing cargo) has been promoted in main in LP#1993819

LP#1990655 and LP#1991650 have MIR Team and Security ACK for main and are cargo dependencies.

Revision history for this message
Didier Roche-Tolomelli (didrocks) wrote :

> This seems ready for re-promotion, now. After bug #1993819 bug #1991650 and bug #1990655 got resolved.

As they are all ACKed now, setting the status back to fix committed. It can be set to fix released with the others once it’s promoted.

Changed in rustc (Ubuntu):
status: New → Fix Committed
Lukas Märdian (slyon)
Changed in rustc (Ubuntu):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.