dpkg 1.22.0ubuntu1 breaking changes

[Mark Esler]

dpkg added new compiler flags in 1.22.0ubuntu1 [0][1] which have caused misbuilt packages.

Two known cases are qemu and dovecot.

qemu was fixed in 1:8.04+dfsg-1ubuntu2 [2] by correcting architecture dependencies (-fcf-protection is only meant for certain x86 archs).

Please note that -fcf-protection is incompatible with -mindirect-branch. Most packages which use -mindirect-branch were likely addressed when -fcf-protection was introduced in 19.10 [3]. Debian is likely more affected in this regard.

For dovecot (LP#2036268) [4], the source of the issue is the dependency libunwind is misbuilt when `-mbranch-protection=standard` is used. libunwind builds, but fails tests when built with this flag on arm64 [5].

Looking at codesearch [6] there are likely many packages affected by libunwind, which may not FTBFS but are misbuilt. There are likely other dependencies, besides libunwind, that also misbuild.

Identifying these regressions in each package is laborious and adds long tail labor. If we can identify batches of misbuilds (like libunwind dependencies) we can avoid excess work and fix packages promptly. Some misbuilds will FTBFS and others will fail tests silently.

dpkg's new compiler flags offer security protections to the Ubuntu Archive and should not be reverted. I suggest that we identify regressions caused by recent dpkg sooner than later. I do not know the scale of affected packages, but this may warrant expensive archive rebuilds which are ran with and without recent dpkg changes.

[0] https://launchpad.net/ubuntu/+source/dpkg/1.22.0ubuntu1
[1] https://git.dpkg.org/cgit/dpkg/dpkg.git/commit/?id=8f5aca71c1435c9913d5562b8cae68b751dff663
[2] https://launchpad.net/ubuntu/+source/qemu/1:8.0.4+dfsg-1ubuntu2
[3] https://wiki.ubuntu.com/ToolChain/CompilerFlags#A-fcf-protection
[4] https://bugs.launchpad.net/ubuntu/+source/dovecot/+bug/2036268
[5] https://github.com/libunwind/libunwind/issues/647
[6] https://codesearch.debian.net/search?q=libunwind&literal=1

[Mark Esler (eslerm)]

Debian devs might be interested in https://bugs.launchpad.net/ubuntu/+source/gcc-11/+bug/1940029

[Steve Langasek (vorlon)]

I asked for this bug to be opened so there was a public record of the issue. However, I believe we are going to want to treat this as "wontfix" for dpkg and not revert the change; instead working to locate and address the consequent misbuilds before noble release.

[Steve Langasek (vorlon)]

Mark, you had suggested we would be able to detect misbuilds from build logs. Can you elaborate what that would look like, in the case of a successful build that is misbuilt?

[Mark Esler (eslerm)]

Specific build flags _might_ have unique error, warning, or failure strings that show up in the log. I could not find any such unique string in libunwind logs built with different -mbranch-protection flags. From the u-boot log [0] `error: ‘-fcf-protection=full’ is not supported for this target` looks like a unique string to search recent archive logs with.

It makes some sense that the stack unwind library conflicts with mbranch-protection. I'll take that as a positive sign that the impact is low for this flag :)

Anything with a vendored version of libunwind needs triage. The libunwind package has not rebuilt binaries recently [1], when it rebuilds it will misbuild and impact reverse dependencies.

I had hoped that the 19.10 introduction of -fcf-protection [2] would have lightened impact, but new problems are popping up [0]. Re-triaging old `-fcf\-protection` LP bug reports seems worthwhile. Retpoline or other uses of -mindirect-branch with -fcf-protection should be triaged [3].

Searching bug trackers from other distros might reveal packages worth triaging.

[0] https://bugs.launchpad.net/ubuntu/+source/u-boot/+bug/2034536
[1] https://bugs.launchpad.net/ubuntu/+source/libunwind/+bug/2041694
[2] https://bugs.launchpad.net/ubuntu/+source/gcc-11/+bug/1940029
[3] https://bugs.launchpad.net/ubuntu/+source/virtualbox/+bug/1835764