ELF package metadata failure: environment variable ‘DEB_HOST_ARCH’ not defined

[Benjamin Drung]

The ELF package note metadata introduced in dpkg 1.22.6ubuntu11 (refined in 1.22.6ubuntu14) can cause this failure:

```
gcc fatal error: environment variable ‘DEB_HOST_ARCH’ not defined
```

This happens when the `-specs=/usr/share/dpkg/elf-package-metadata.specs` parameter is set but the needed environment variables are not set. Cases:

1. Only the LDFLAGS is queried from dpkg-buildflags. Affected source packages builds: python3.12, openjdk-21, cdbs (causing dvbstreamer and rp-pppoe fail to build)

2. autopkgtests

3. debugging a crash of an application build with the -spec parameter

4. People like to iteratively continue building the software in the build dir while hacking on the package and then have no environment variable set.

This approach is too fragile. An alternative approach would be to specify the `--package-metadata` linker flag directly. The problem with that approach is that the curly brackets and quotation marks need to be escaped. Example failure: Building dpkg would add this parameter to the LDFLAGS:
```
-Wl,--package-metadata,{"type":"deb","os":"ubuntu","name":"dpkg","version":"1.22.6ubuntu15","architecture":"amd64"}
```
The following configure script call (non-relevant parameters deleted):
```
$ gcc -Wl,--package-metadata,{type:deb,os:ubuntu,name:dpkg,version:1.22.6ubuntu15,architecture:amd64}
/usr/bin/ld: cannot find {type:deb: No such file or directory
/usr/bin/ld: cannot find os:ubuntu: No such file or directory
/usr/bin/ld: cannot find name:dpkg: No such file or directory
/usr/bin/ld: cannot find version:1.22.6ubuntu15: No such file or directory
/usr/bin/ld: cannot find architecture:amd64}: No such file or directory
```

Proposed solution: Add support for an `--escaped-package-metadata` parameter to the linkers that takes an URL encoded (RFC 3986) parameter. Example:
```
-Wl,--encoded-package-metadata,%7B%22type%22:%22deb%22%2C%22os%22:%22ubuntu%22%2C%22name%22:%22dpkg%22%2C%22version%22:%221.22.6ubuntu15%22%2C%22architecture%22:%22amd64%22%7D
```

[Luca Boccassi (bluca)]

> Proposed solution: Add support for an `--escaped-package-metadata` parameter to the linkers that takes an URL encoded (RFC 3986) parameter.

It took months and a huge amount of fighting to get the option merged in all linkers, it was extremely hard. I am afraid this is just unrealistic.

The opt-out is there exactly to skip problematic packages, I recommend to just use it and skip those packages that are not doing standard debhelper builds. It's a small percentage nowadays anyway, so it's fine to just skip them.

[Matthias Klose (doko)]

just curious: are there any pointers to these discussions? It seems odd to design a format that has quoting issues from the beginning.

I don't like having a 99% solution, and skip stuff otherwise. Plus it's not just problematic for regular package builds, but also for debugging a package build. You have to set all these environment variables on your own, when you restart some part of a package build.

I see two ways to get around that:

 - a package build creates a build specific spec without relying on any environment variables. That would be a change in dpkg, plus passing the --package-metadata in the build flags.

 - the gcc and clang drivers insert the --package-metadata option on it's own, if it is not present, and if all the env vars are present. This would not need *any* changes in dpkg, just a simple local patch to the compiler drivers. In case of not having the env vars, it just would not be recorded, but not failing the build as it's currently the case.

Both options seem to be better than the current status.

[Luca Boccassi (bluca)]

> just curious: are there any pointers to these discussions? It seems odd to design a format that has quoting issues from the beginning.

Unfortunately I don't have links, it's across various maling lists and bug trackers from 2/3 years ago or so.

> I don't like having a 99% solution, and skip stuff otherwise.

Every dpkg feature/flag is a 99% solution. Hardening, frame pointers, reproducibility, none of these feature flags have 100% usage across the entire package base - and that's normal, because there's always a 1% of packages doing really, really weird stuff, and it's just not worth the effort. This already approaches max utility with a 99% coverage, there is very very little to be gained to go out of our way to cover the remaining 1%.

> - a package build creates a build specific spec without relying on any environment variables. That would be a change in dpkg, plus passing the --package-metadata in the build flags.

That gets messy because you have to create the file and manage it and delete it, at exactly the right time, and again against packages that will do extremely weird things to any location that you might pick to write it out to. Pick any directory in the build tree or outside it, and there will be at least one package rm -rf'ing it at the wrong time. We tried this first in Fedora, and then quickly moved away from it, as it was just too difficult and much more prone to failure.

> - the gcc and clang drivers insert the --package-metadata option on it's own, if it is not present, and if all the env vars are present.

This requires again changing all the linkers, and it is extremely unlikely that such changes will be accepted, so you'll have to carry out of tree patches forever for bfd, gold, lld and mold.

If you want to change the linkers, it seems to me it would be much more likely to be able to land a patch that changes the spec file's getenv() to take a default fallback value in case the env var doesn't exist, or something along those lines. Then it would just set the value to an empty string when missing, and that will be enough to stop the build failures.

Alternatively, originally the change did not use dpkg-buildflags to add to ldflags. It could be changed again to avoid that, if the problem is weird packages using dpkg-buildflags manually and only picking some bits from it. I could go back to being set via DEB_LDFLAGS_MAINT_APPEND for example.

[Andreas Hasenack (ahasenack)]

The mwrap tests[1] are failing with the error[2] reported in this bug.

1. https://autopkgtest.ubuntu.com/packages/mwrap/oracular/amd64
2. https://autopkgtest.ubuntu.com/results/autopkgtest-oracular/oracular/amd64/m/mwrap/20240629_173552_2116e@/log.gz

[Andreas Hasenack (ahasenack)]

Here is a list of current DEP8 failures with the error message from this bug:

$ ./retry-autopkgtest-regressions --log-regex "fatal error: environment variable ‘DEB_HOST_ARCH’ not defined"
https://autopkgtest.ubuntu.com/request.cgi?release=oracular&arch=amd64&package=asymptote&trigger=dpkg%2F1.22.6ubuntu14
https://autopkgtest.ubuntu.com/request.cgi?release=oracular&arch=arm64&package=asymptote&trigger=dpkg%2F1.22.6ubuntu14
https://autopkgtest.ubuntu.com/request.cgi?release=oracular&arch=ppc64el&package=asymptote&trigger=dpkg%2F1.22.6ubuntu14
https://autopkgtest.ubuntu.com/request.cgi?release=oracular&arch=amd64&package=mosh&trigger=dpkg%2F1.22.6ubuntu14
https://autopkgtest.ubuntu.com/request.cgi?release=oracular&arch=arm64&package=mosh&trigger=dpkg%2F1.22.6ubuntu14
https://autopkgtest.ubuntu.com/request.cgi?release=oracular&arch=ppc64el&package=mosh&trigger=dpkg%2F1.22.6ubuntu14
https://autopkgtest.ubuntu.com/request.cgi?release=oracular&arch=s390x&package=mosh&trigger=dpkg%2F1.22.6ubuntu14

asymptote and mosh (why it didn't pick up my mwrap run from the previous comment I don't know)

[Jeremy Bícha (jbicha)]

gsequencer failed to build when I tried a test build locally. :(

[Launchpad Janitor (janitor)]

This bug was fixed in the package mosh - 1.4.0-1ubuntu4

---------------
mosh (1.4.0-1ubuntu4) oracular; urgency=medium

  * Fix autopkgtests following dpkg changes (LP: #2071468)

 -- Jeremy Bícha <email address hidden> Sat, 29 Jun 2024 19:34:41 -0400

[Launchpad Janitor (janitor)]

This bug was fixed in the package mwrap - 1.1.1-2ubuntu1

---------------
mwrap (1.1.1-2ubuntu1) oracular; urgency=medium

  * Fix autopkgtests following dpkg changes (LP: #2071468)

 -- Jeremy Bícha <email address hidden> Sat, 29 Jun 2024 18:59:50 -0400

[Launchpad Janitor (janitor)]

This bug was fixed in the package asymptote - 2.89+ds-1ubuntu1

---------------
asymptote (2.89+ds-1ubuntu1) oracular; urgency=medium

  * Fix autopkgtest following dpkg changes (LP: #2071468)

 -- Jeremy Bícha <email address hidden> Sat, 29 Jun 2024 20:29:47 -0400

[Launchpad Janitor (janitor)]

This bug was fixed in the package gsequencer - 6.13.5-1ubuntu1

---------------
gsequencer (6.13.5-1ubuntu1) oracular; urgency=medium

  * Fix autopkgtests following dpkg changes (LP: #2071468)

 -- Jeremy Bícha <email address hidden> Sun, 30 Jun 2024 08:36:20 -0400

[Athos Ribeiro (athos-ribeiro)]

Squid tests are also affected.

[Benjamin Drung (bdrung)]

The failing mwrap did not set the LDFLAGS on its own. `strings /usr/bin/mkoctfile` shows that mkoctfile includes `-specs=/usr/share/dpkg/elf-package-metadata.specs`.

[Vladimir Petko (vpa1977)]

libreoffice autopkgtests fail with the following error:

871s S=/tmp/autopkgtest.YSUwcU/build.7pH/src && I=$S/instdir && W=$S/workdir && x86_64-linux-gnu-gcc -Wl,-z,origin '-Wl,-rpath,$ORIGIN/../Library' -Wl,-rpath-link,$I/program -Wl,-z,defs -Wl,-rpath-link,/lib:/usr/lib -Wl,-z,combreloc -Wl,--hash-style=gnu -Wl,-Bsymbolic-functions -L$W/LinkTarget/StaticLibrary -L$I/sdk/lib -L$I/program -L$I/program -L$W/LinkTarget/Library -Wl,-Bsymbolic-functions -Wl,-z,relro -specs=/usr/share/dpkg/elf-package-metadata.specs $W/CObject/solenv/bin/concat-deps.o -Wl,--start-group -Wl,--end-group -Wl,--no-as-needed -o $W/LinkTarget/Executable/concat-deps
871s x86_64-linux-gnu-gcc: fatal error: environment variable ‘DEB_HOST_ARCH’ not defined

This is blocking openjdk-21 migration

[1] https://objectstorage.prodstack5.canonical.com/swift/v1/AUTH_0f9aae918d5b4744bf7b827671c86842/autopkgtest-oracular/oracular/amd64/libr/libreoffice/20240704_005705_46cf5@/log.gz

[Launchpad Janitor (janitor)]

This bug was fixed in the package squid - 6.9-1ubuntu1

---------------
squid (6.9-1ubuntu1) oracular; urgency=medium

  * Merge with Debian unstable (LP: #2064466). Remaining changes:
    - d/usr.sbin.squid: Add sections for squid-deb-proxy and
      squidguard
    - d/p/90-cf.data.ubuntu.patch: Add refresh patterns for deb
      packaging
    - Use snakeoil certificates:
      + d/control: add ssl-cert to dependencies
      + d/p/99-ubuntu-ssl-cert-snakeoil.patch: add a note about ssl
        to the default config file
    - d/NEWS: drop the NIS basic auth helper (LP #1895694)
    - d/p/0009-Fix-Werror-alloc-size-larger-than-on-GCC-12.patch:
      Fix FTBFS due to -Werror=alloc-size-larger-than on GCC 12.
    - d/rules: halt build upon test failures.
    - d/rules: do not include additional configuration files during
      build time tests. This would lead to test failures due to missing
      paths.
    - d/t/upstream-test-suite: use installed squid binary for
      autopkgtest config file checks.
    - d/p/0010-Fix-Werror-sign-compare-on-GCC-13.patch: fix comparison
      between signed and unsigned values.
    - d/rules: disable LTO related compilation errors for ppc64el builds.
    - d/source_squid.py, d/squid-common.install: Add apport hook
      (LP #676141)
  * Dropped changes:
    - SECURITY UPDATE: DoS via chunked decoder uncontrolled recursion bug
      + debian/patches/CVE-2024-25111.patch: fix infinite recursion in
        src/http.cc, src/http.h.
      + CVE-2024-25111
      [ Fixed in 6.8 ]
  * New changes:
    - d/t/upstream-test-suite: adjust autopkgtests following dpkg
      changes enabling ELF metadata. (LP: #2071468)

squid (6.9-1) unstable; urgency=medium

  [ Amos Jeffries <email address hidden> ]
  * New Upstream Release 6.9

squid (6.8-1) unstable; urgency=high

  [ Amos Jeffries <email address hidden> ]
  * New Upstream Release 6.8
    Fixes: CVE-2024-25111. SQUID-2024:1

  [ Luigi Gangitano <email address hidden> ]
  * debian/control
    - Migrate from pkg-config to pkgconf

 -- Athos Ribeiro <email address hidden> Tue, 02 Jul 2024 14:21:39 -0300

[Launchpad Janitor (janitor)]

This bug was fixed in the package proftpd-dfsg - 1.3.8.b+dfsg-2ubuntu1

---------------
proftpd-dfsg (1.3.8.b+dfsg-2ubuntu1) oracular; urgency=medium

  * d/t/test-suite: Add workaround to fix LP: #2071468.

 -- Sergio Durigan Junior <email address hidden> Mon, 08 Jul 2024 16:31:41 -0400

[Gianfranco Costamagna (costamagnagianfranco)]

swtpm tests are also affected

[Launchpad Janitor (janitor)]

This bug was fixed in the package swtpm - 0.7.3-0ubuntu6

---------------
swtpm (0.7.3-0ubuntu6) oracular; urgency=medium

  * Fix autopkgtests following dpkg changes (LP: #2071468)

 -- Gianfranco Costamagna <email address hidden> Tue, 09 Jul 2024 12:29:58 +0200

[Gianfranco Costamagna (costamagnagianfranco)]

haskell-devscripts also affected

[Launchpad Janitor (janitor)]

This bug was fixed in the package haskell-devscripts - 0.16.33ubuntu2

---------------
haskell-devscripts (0.16.33ubuntu2) oracular; urgency=medium

  * Include langcore.mk, this fixes FTBFS due to changes in dpkg and cdbs 0.4.166ubuntu1
    (LP: #2071468)

 -- Gianfranco Costamagna <email address hidden> Tue, 09 Jul 2024 17:26:03 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package serf - 1.3.10-3ubuntu1

---------------
serf (1.3.10-3ubuntu1) oracular; urgency=medium

  * Fix build following dpkg changes (LP: #2071468)

 -- Gianfranco Costamagna <email address hidden> Mon, 08 Jul 2024 10:15:28 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package curl - 8.8.0-3ubuntu3

---------------
curl (8.8.0-3ubuntu3) oracular; urgency=medium

  * d/{control,rules}: Do not use gnutls for the curl binary
  * d/{control,rules}: Drop nghttp3 & ngtcp2 depdendencies of libcurl-gnutls
    to avoid component-mismatch

 -- Lukas Märdian <email address hidden> Tue, 09 Jul 2024 14:30:56 +0200