grub-efi-arm64-signed : Depends: grub-efi-arm64 (= 2.06-2ubuntu7) but 2.06-2ubuntu10 is to be installed

[elatllat]

[Impact]
Phased updates with interdependencies can become uninstallable, breaking the upgrade command (and presumably dist-upgrade), so it cannot upgrade
anything anymore, for example, see the original bug report below.

[Test plan]
We cannot reproduce the exact failure here, but we have included a minimal test case that triggers the same error in the integration test test-phased-updates-upgrade.

The test suite is exhaustive and covers more than a human reasonably can, especially since we also cannot publish broken updates again to get people to test the fix.

[Where problems could occur]
This is a one word fix in the solver. When deciding whether a depends could be kept back, we accidentally looked at the package declaring the dependency instead of the package we depend upon.

Of course this might unearth other fancy solver bugs. There is one
limitation in that this only applies to "Protected" packages (not in the Protected field sense, but if you "Protect" the change in the solver), which is *mostly* triggered only if you have held packages or phased updates, so workarounds should always be easy.

[Original bug report]

# apt clean

# apt update
Hit:1 http://ca-central-1.ec2.ports.ubuntu.com/ubuntu-ports jammy InRelease
Hit:2 http://ca-central-1.ec2.ports.ubuntu.com/ubuntu-ports jammy-updates InRelease
Hit:3 http://ca-central-1.ec2.ports.ubuntu.com/ubuntu-ports jammy-backports InRelease
Hit:4 http://ports.ubuntu.com/ubuntu-ports jammy-security InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
24 packages can be upgraded. Run 'apt list --upgradable' to see them.

# apt upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
 grub-efi-arm64-signed : Depends: grub-efi-arm64 (= 2.06-2ubuntu7) but 2.06-2ubuntu10 is to be installed
E: Broken packages

# apt show -a grub-efi-arm64 2>/dev/null | grep -P "Version:|APT-Sources:"
Version: 2.06-2ubuntu10
APT-Sources: http://ca-central-1.ec2.ports.ubuntu.com/ubuntu-ports jammy-updates/main arm64 Packages
Version: 2.06-2ubuntu7
APT-Sources: http://ca-central-1.ec2.ports.ubuntu.com/ubuntu-ports jammy/main arm64 Packages

# curl -s "http://ports.ubuntu.com/ubuntu-ports/dists/jammy/main/binary-arm64/Packages.gz" | zgrep -B 13 "Depends: grub-efi-arm64" | grep -P "Package:|Depends:"
Package: grub-efi-arm64-dbg
Depends: grub-efi-arm64-bin (= 2.06-2ubuntu7)
Package: grub-efi-arm64-signed
Depends: grub-efi-arm64 (= 2.06-2ubuntu7)

# curl -s "http://ports.ubuntu.com/ubuntu-ports/dists/jammy-updates/main/binary-arm64/Packages.gz" | zgrep -B 13 "Depends: grub-efi-arm64" | grep -P "Package:|Depends:"
Package: grub-efi-arm64-dbg
Depends: grub-efi-arm64-bin (= 2.06-2ubuntu10)
Package: grub-efi-arm64-signed
Depends: grub-efi-arm64 (= 2.06-2ubuntu10)

[[Workaround]]
echo "APT::Get::Never-Include-Phased-Updates: 1;" > /etc/apt/apt.conf.d/99phased-updates

ProblemType: Bug
DistroRelease: Ubuntu 22.04
Package: grub-efi-arm64-signed 1.180+2.06-2ubuntu7
ProcVersionSignature: Ubuntu 5.15.0-1020.24-aws 5.15.53
Uname: Linux 5.15.0-1020-aws aarch64
ApportVersion: 2.20.11-0ubuntu82.1
Architecture: arm64
CasperMD5CheckResult: unknown
Date: Fri Sep 23 13:12:07 2022
Ec2AMI: ami-092d30890417a55e5
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: ca-central-1d
Ec2InstanceType: t4g.medium
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
ProcEnviron:
 SHELL=/bin/bash
 LANG=C.UTF-8
 TERM=xterm-256color
 XDG_RUNTIME_DIR=<set>
 PATH=(custom, no user)
SourcePackage: grub2-signed
UpgradeStatus: No upgrade log present (probably fresh install)

[Julian Andres Klode (juliank)]

Please include output of

 apt policy grub-efi-arm64 grub-efi-arm64-signed

And make sure you do not have any preferences file blocking the upgrade.

This might be caused by the same phasing issue as for python3.10:

[Bug 1990586] [NEW] 22.04: python3* mismatched phasing breaks dist-upgrades

But it also might be a configuration issue.

[Steve Langasek (vorlon)]

This is a bug in how apt handles phased updates across multiple source packages currently. The grub SRU has been rolled back to jammy-proposed now, and an apt fix is in progress, expected to be uploaded early next week.

[elatllat (elatllat)]

Thanks Steve, is there a workaround before then?

Julian;

apt policy grub-efi-arm64 grub-efi-arm64-signed
grub-efi-arm64:
  Installed: 2.06-2ubuntu7
  Candidate: 2.06-2ubuntu10
  Version table:
     2.06-2ubuntu10 500 (phased 13%)
        500 http://ca-central-1.ec2.ports.ubuntu.com/ubuntu-ports jammy-updates/main arm64 Packages
 *** 2.06-2ubuntu7 500
        500 http://ca-central-1.ec2.ports.ubuntu.com/ubuntu-ports jammy/main arm64 Packages
        100 /var/lib/dpkg/status
grub-efi-arm64-signed:
  Installed: 1.180+2.06-2ubuntu7
  Candidate: 1.182~22.04.1+2.06-2ubuntu10
  Version table:
     1.182~22.04.1+2.06-2ubuntu10 500 (phased 13%)
        500 http://ca-central-1.ec2.ports.ubuntu.com/ubuntu-ports jammy-updates/main arm64 Packages
 *** 1.180+2.06-2ubuntu7 500
        500 http://ca-central-1.ec2.ports.ubuntu.com/ubuntu-ports jammy/main arm64 Packages
        100 /var/lib/dpkg/status

[elatllat (elatllat)]

For the record

    echo -e "Update-Manager::Never-Include-Phased-Updates;\nAPT::Get::Never-Include-Phased-Updates: True;" > /etc/apt/apt.conf.d/99phased-updates

did not change the buggy behaviour

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[elatllat (elatllat)]

grub-efi-arm64 is not listed here;

https://people.canonical.com/%7Eubuntu-archive/phased-updates.html

Maybe that page is limited to x86_64.

[Carlos F. (acsfer)]

Set:

APT::Get::Never-Include-Phased-Updates "1";

and try again. Packages will be kept back.

...
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages have been kept back:
  grub-efi-arm64 grub-efi-arm64-bin grub-efi-arm64-signed libnss-systemd libpam-systemd libsystemd0 libudev1 systemd
  systemd-sysv systemd-timesyncd udev
The following packages will be upgraded:
...

[elatllat (elatllat)]

Thanks Carlos, I now have a workaround;

    echo "APT::Get::Never-Include-Phased-Updates: 1;" > /etc/apt/apt.conf.d/99phased-updates

[Julian Andres Klode (juliank)]

This issue seems to have been a funny bug in the solver, actually, and not a bug in the phased update code itself.

So what happened is that we kept back grub-efi-amd64-signed, and it was broken. So now the solver iterates over its dependencies and tries to keep them back too. Except well, when looking at whether the dependency could be kept back, it checked if parent package - grub-efi-amd64-signed - was protected and not the dependency.

Hilarious, but fixed in

https://salsa.debian.org/apt-team/apt/-/merge_requests/262/diffs?commit_id=ac2a5196326388b3a0a60090fe787b1eb85ed22a

with a simplified test case.

[Robie Basak (racb)]

In terms of risk, touching apt is always going to be high risk because users need apt to revert or upgrade away from any regression. QA is therefore essential. Normally I'd like this to "bake" in the development release first, but given the nature of the bug, that's probably not going to give us any practical benefit over testing this in jammy-proposed. In terms of the details of testing, I can't think of anything that we could do better over relying on the test suite, and I'm happy to see that these bugfixes come with corresponding tests.

Since Kinetic is in beta freeze but this bugfix is urgent, I think it makes sense to accept this into jammy-proposed immediately to maximise testing opportunity.

[Robie Basak (racb)]

Hello elatllat, or anyone else affected,

Accepted apt into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/2.4.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[elatllat (elatllat)]

Robie; thanks for the testing instructions link.

verification-done-jammy;

This bug is testable on AWS EC2 Graviton 1/2/3 (c6g,g5g,r6g,t4g) the cheapest of which is the t4g.nano ranging from 0.00280 0.00670 $USD/h depending on the region. So that's what I did.

# echo -e "APT::Get::Always-Include-Phased-Updates: 1;" > /etc/apt/apt.conf.d/99phased-updates

# apt clean && apt update && apt upgrade

# apt show -a grub-efi-arm64 2>/dev/null | grep -P "Version:|APT-Sources:"
Version: 2.06-2ubuntu7
APT-Sources: http://ca-central-1.ec2.ports.ubuntu.com/ubuntu-ports jammy/main arm64 Packages

# echo "\"...The grub SRU has been rolled back...\" passed the test because only the old version was listed"

# cat <<EOF >/etc/apt/sources.list.d/ubuntu-$(lsb_release -cs)-proposed.list
# # Enable Ubuntu proposed archive
# deb http://ports.ubuntu.com/ubuntu-ports $(lsb_release -cs)-proposed restricted main multiverse universe
# EOF

# apt clean && apt update && apt upgrade

# apt show -a grub-efi-arm64 2>/dev/null | grep -P "Version:|APT-Sources:"
Version: 2.06-2ubuntu10
APT-Sources: http://ports.ubuntu.com/ubuntu-ports jammy-proposed/main arm64 Packages
Version: 2.06-2ubuntu7
APT-Sources: http://ca-central-1.ec2.ports.ubuntu.com/ubuntu-ports jammy/main arm64 Packages

# echo "\"... to jammy-proposed...\" passed the test because both versions were listed"

# shutdown -r now

# apt show grub-efi-arm64 2>/dev/null | grep Version
Version: 2.06-2ubuntu10

# echo "\"...fixed...\" passed the test because the new version is in use, apt worked, and reboot worked."

[Julian Andres Klode (juliank)]

elatllat no that's non-sense on multiple levels

a) the grub update is not phasing anymore so it's not reproducing the bug, we bumped it back to proposed. There's no phasing there so it just works anyway
b) setting the option in the config file would work around the bug too. Of course you added a trailing : to the name that doesn't belong there so it's not doing anything.

Reverting the tags for now for log purposes.

[Julian Andres Klode (juliank)]

It seems the autopkgtest runs have finished for 2.4.8 as well, and all passed, so marking as verified once again.

For those stalking the test logs,

(239/290) Testcase test-phased-updates-upgrade: W: Ignoring failure of dpkg --add-architecture as it is likely due to pkg-config-dpkghook (see #824774)
W: Ignoring failure of dpkg --add-architecture as it is likely due to pkg-config-dpkghook (see #824774)
 P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P

This was the test that includes the test case for this bug.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 2.5.3

---------------
apt (2.5.3) unstable; urgency=medium

  [ Povilas Kanapickas ]
  * doc: Explain that apt-get download ensures package authenticity

  [ David Kalnischkies ]
  * Keep color disabled if APT::Color was set to false
  * Don't reuse filename for in-tree builds of apt
  * Respect users pkg order on `apt install` for resolving

  [ Hideki Yamane ]
  * Fix typo in Japanese program translation

  [ lancethepants ]
  * porting (musl):
    - apt-pkg/contrib/fileutl.h Explicitly include sys/stat.h

  [ Khem Raj ]
  * porting (clang15):
    - typecast time_t and suseconds_t from std::chrono
    - interactive-helper: Undefine _FORTIFY_SOURCE

  [ Julian Andres Klode ]
  * edsp: Add support for phased updates (test in LP#1990586)
  * Check state of dependency, not depender in dependency keep back
    (LP: #1990684)
  * full-upgrade: Mark phased upgrades for keep before anything else
    (LP: #1990586)

 -- Julian Andres Klode <email address hidden> Wed, 28 Sep 2022 17:16:41 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 2.4.8

---------------
apt (2.4.8) jammy; urgency=medium

  * edsp: Add support for phased updates (test in LP#1990586)
  * Check state of dependency, not dependee in dependency keep back
    (LP: #1990684)
  * full-upgrade: Mark phased upgrades for keep before anything else
    (LP: #1990586)

 -- Julian Andres Klode <email address hidden> Wed, 28 Sep 2022 17:37:51 +0200

[Timo Aaltonen (tjaalton)]

The verification of the Stable Release Update for apt has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.