apt-get update should return exit code != 0 on error

[Jens Elkner]

[Impact]
There is no way for scripts to run apt update and be sure that it was entirely successful.

We introduce a new flag, --error-on=any that makes apt update also error out on transient errors.

[Test plan]
We have included a test in the test-suite in test-apt-update-failure-propagation that ensures that warnings become errors.

[Where problems could occur]
If there's an issue some warnings for transient errors could become errors, or worse I suppose it's possible but unlikely for errors to be reported as warnings...

Since the new behavior is behind a flag, regression potential for existing scripts is low if there is no bug.

[Original bug report]

When running 'apt-get update' (e.g. on a container install post-install script), apt-get return with exit code 0, even so it wasn't able to "update" properly. E.g.:

+ apt-get update
Err:1 http://de.archive.ubuntu.com/ubuntu xenial InRelease
  Temporary failure resolving 'de.archive.ubuntu.com'
Err:2 http://security.ubuntu.com/ubuntu xenial-security InRelease
  Temporary failure resolving 'security.ubuntu.com'
Err:3 http://de.archive.ubuntu.com/ubuntu xenial-updates InRelease
  Temporary failure resolving 'de.archive.ubuntu.com'
Reading package lists... Done
W: Failed to fetch http://de.archive.ubuntu.com/ubuntu/dists/xenial/InRelease Temporary failure resolving 'de.archive.ubuntu.com'
W: Failed to fetch http://de.archive.ubuntu.com/ubuntu/dists/xenial-updates/InRelease Temporary failure resolving 'de.archive.ubuntu.com'
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/xenial-security/InRelease Temporary failure resolving 'security.ubuntu.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.

It should be corrected to return useful exit code, so that scripts can take the appropriate actions ...

[Julian Andres Klode (juliank)]

Well, it did not fail, it re-used the existing information. Transient network errors do not cause the update to fail, because they are expected to be temporary. It's always been like this. I'm open to possibly changing this, but such a change could cause huge issues.

[Jens Elkner (jelmd)]

Hmmm, IMHO 'Err:' and 'W: Failed to fetch' indicate, that it was not able to update properly and thus the result of the operation is unreliable. So its like "your connection is secured, but may be not". I guess most people wouldn't do any financial transaction when reading this ... ;-)

[Julian Andres Klode (juliank)]

Well, it's more like "this site is temporarily unavailable, please try again at a later time."

[Jarno Suni (jarnos)]

Because of this bug ppa-purge misbehaves (Bug #1514839). Should one inspect the output of apt-get for '^Err:' (from standard output) or even '^W:' (from standard error) to determine, if it updated the repositories or not? Besides the output of `apt-get update` varies between versions in supported Ubuntu releases: In Ubuntu 14.04 it does not print ':' after 'Err', but ' ', instead.

[Julian Andres Klode (juliank)]

You should probably use an API for that (like python-apt), rather than the command-line tools. We will have to rework error handling if we ever want to retry automatic updates on failures, but it's not entirely clear how this will happen yet.

In any case, I do consider an update where not all repositories were updated a success. But I think some repository should update.

[Jens Elkner (jelmd)]

The bug/subject here is, that apt-get doesn't return a proper exit code, not, what else one could use to workaround the bug.

Saying, that update from one of all (i.e. 1+) sites is sufficient is like going to fly with a jet, where one of two engines is already broken before it starts. Making this decision for yourself is ok, but making it for all other passengers is IMHO irresponsible.

[Julian Andres Klode (juliank)]

No, there are two different use cases here:

1. Did apt-get update (potentially) change something?
2. Did apt-get update (potentially) change everything?

In the general case, 1 is what you need to know. For example, a tool that runs update and then upgrades your system: You don't want to prevent the system from being upgraded (in an update&&upgrade type scenario) just because some third party repository disappeared.

Then there is the question of fatal vs non-fatal errors. A DNS resolution error like here is most likely temporary, you don't want to fail update because of it, this would just confuse users (don't spam me twice a day with errors just because my machine is offline).

[Jens Elkner (jelmd)]

Actually this is the problem: Users think, their system is up-to-date, but it is not for sure because a site failed to respond. Therefore only if _all_ sites answered the request properly, apt-get should return 0. If not, it should return a specified return code, which lets the callee know, that there was a problem [and imply, that a subsequent apt-get upgrade might bring the system to the latest supported state, or not]. If the exit code for such situations is documented properly, the tool can still decide, whether to run the upgrade or would be bredless art.

BTW: It doesn't really matter, what error (whether temp. DNS, LDAP lookup, network, etc.) prevented a successful update. Fact is, that there was an error and this needs to be communicated. CLI tools do that via exit code.

[Julian Andres Klode (juliank)]

That's your opinion, and we already know it.

Whether we want to do it that or if it is technically feasible are different questions. Fact is: Our results are boolean, we might not be able to really influence the exit status.

I don't want to discuss this with you, I was just giving jarnos a hint how ppa-purge could possibly be fixed.

[Jarno Suni (jarnos)]

juliank, Do you mean ppa-purge should be translated to completely different programming language, and that I can not even rely on apt-get outputting something that matches '^Err[ :]' to stdout in such cases? Sounds difficult.

What do you mean by "Our results are boolean, we might not be able to really influence the exit status." I do not understand why the "results" could not be changed to be something else.

You say: "You don't want to prevent the system from being upgraded (in an update&&upgrade type scenario) just because some third party repository disappeared."
Something like
'apt-get update || [ $? -ne "$FATAL_APT_UPDATE_ERROR" ] && apt-get upgrade'
in command line would let upgrade happen even in case of partially successful update.

[Julian Andres Klode (juliank)]

I can only tell you that either python or C++ should give you what you need at the moment. You could potentially just parse Err lines, but that's a bit hacky. As you mentioned, the output changed a bit.

Our functions return booleans every in the stack, it's unclear how and if we can change that to something else in the near future. I was thinking more about specifying the goal you want (update something, update all, etc.) and then failing if the goal is not reached.

[Ryan Jaeb (ryanjaeb)]

@Julian Andres Klode Is this something you'd be willing to reassess? I saw your old message [1] on the Debian list that says:

> The question what a successful update is is complicated and depends
on the expections of the person using APT.

With that in mind, I'd say there are situations where it's reasonable for the expectation to be that `apt-get update` fails on any error. I completely understand the reasoning for the default behavior and don't expect that to change, but a `--strict` switch (or similar) like the one suggested in the mailing list would be extremely useful to some of us.

Even an advanced option like (note this is not real) `-o=APT::Get::Strict-Mode=1` would be useful if that's easier than a full blown command line switch. I'm not sure I got the syntax right, but I'm sure you get the idea.

With containerization gaining popularity, I think there could be a lot of people who have automated build systems set up where they pull a container, (try to) apply security updates, and so on... In those cases, the warnings are really hard to notice since the build system claims everything is OK. Automated build systems that use Docker tend to surface the problem a bit more because of the way they handle DNS [2][3]. The TLDR of those issues is that Docker tries to be smart and sometimes falls back to Google DNS:

> level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"

For anyone using an internal APT mirror, a bunch of assumptions get broken, but nothing throws an error to surface the break down. I bet there's a subset of developers that don't realize their automated builds could be failing to apply updates.

I spent several hours today trying to find a reasonable workaround, but all I could find were people trying to parse apt-get's output in a variety of creative ways. AFAIK, there aren't any good workarounds. I even tried fumbling my way through the source code to see if there were any existing options to control the behavior, but didn't have much luck (I don't know C++ though).

So, I guess I have two questions. First, is there _any_ way to make `apt-get update` fail on any error? Second, if the answer to my first question is no, would you consider adding an option that lets us control it?

1) https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=776152#15
2) https://github.com/moby/moby/issues/23910
3) https://github.com/moby/libnetwork/issues/1654

[Scott Moser (smoser)]

For reference, ubuntu-devel post about '-o APT::Update::Error-Mode=any'
at https://lists.ubuntu.com/archives/ubuntu-devel/2021-February/041374.html

[Julian Andres Klode (juliank)]

Thanks Scott for digging out this bug again, I'll use it for SRUing.

[Brian Murray (brian-murray)]

Hello Jens, or anyone else affected,

Accepted apt into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/2.1.10ubuntu0.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted apt (2.1.10ubuntu0.3) for groovy have finished running.
The following regressions have been reported in tests triggered by the package:

reprotest/0.7.15 (arm64, s390x)
livecd-rootfs/2.694.3 (amd64, s390x)
dgit/9.11ubuntu1 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/groovy/update_excuses.html#apt

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Łukasz Zemczak (sil2100)]

Hello Jens, or anyone else affected,

Accepted apt into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/2.0.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Jens, or anyone else affected,

Accepted apt into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/1.6.13 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted apt (1.6.13) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

apport/2.20.9-0ubuntu7.23 (amd64, i386)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#apt

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted apt (2.0.5) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

reprotest/0.7.14 (s390x, ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#apt

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Mathew Hodson (mhodson)]

Fixed in Ubuntu Hirsute.

---
apt (2.1.16) unstable; urgency=medium

  [ Faidon Liambotis ]
  * Various fixes to http and connect method
    - basehttp: also consider Access when a Server's URI
    - connect: convert a C-style string to std::string
    - connect: use ServiceNameOrPort, not Port, as the cache key

  [ Julian Andres Klode ]
  * patterns: Add dependency patterns ?depends, ?conflicts, etc.
    Note that the -broken- variants are not implemented yet.
  * Rewrite of the kernel autoremoval code:
    - Determine autoremovable kernels at run-time (LP: #1615381), this fixes the
      issue where apt could consider a running kernel autoremovable
    - Automatically remove unused kernels on apt {full,dist}-upgrade.
      This helps ensuring that we don't run out of /boot space.
    - Only keep up to 3 (not 4) kernels.
      Ubuntu boot partitions were sized for 3 kernels, not 4.
  * Bump codenames to bullseye/hirsute and adjust -security codename for
    bullseye (Closes: #969932)
  * Ignore failures from immediate configuration. This does not change the
    actual installation ordering - we never passed the return code to the
    caller and installation went underway anyway if it could be ordered at a
    later stage, this just removes spurious after-the-fact errors.
    (Closes: #973305, #188161, #211075, #649588) (LP: #1871268)
  * Add support for Phased-Update-Percentage, previously used only by
    update-manager.
  * Implement update --error-on=any so that scripts can reliably check for
    transient failures as well. (Closes: #594813)

  [ Demi M. Obenour ]
  * test/integration/framework: Be compatible with Bash

  [ Vangelis Skarmoutsos ]
  * Greek program translation update

 -- Julian Andres Klode <email address hidden> Fri, 08 Jan 2021 21:49:15 +0100

[Jarno Suni (jarnos)]

I tried the apt provided by focal-proposed. At least
sudo apt-get update -eany
gave exit status 100, if I disabled networking. Without -eany it gave 0.

I got the source code by 'apt source'.
I could not run test-apt-update-failure-propagation:

$ sudo ./apt-2.0.5/test/integration/test-apt-update-failure-propagation
Cannot find build directory, you might want to set PROJECT_BINARY_DIR
E: Looks like the testcases ended prematurely with exitcode: 1

[Julian Andres Klode (juliank)]

The autopkgtests for apt have passed, so testing succeeded. I retried the failures, I expect them to be unrelated: apport one already happened with binutils/2.30-21ubuntu1~18.04.5, the i386 retry succeeded; and reprotest fails in its normal way, a few retries should get it working.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 2.1.10ubuntu0.3

---------------
apt (2.1.10ubuntu0.3) groovy; urgency=medium

  [ David Kalnischkies ]
  * Fix incorrect base64 encoding due to int promotion (LP: #1916050)
  * Harden test for no new acquires after transaction abort (Closes: #984966)
    (LP: #1918920)

  [ Julian Andres Klode ]
  * Implement update --error-on=any (Closes: #594813) (LP: #1693900)
  * Include all translations when building the cache (LP: #1907850)
  * Do not require force-loopbreak on Protected packages (Closes: #983014)
    (LP: #1916725)
  * RunScripts: Do not reset SIGQUIT and SIGINT to SIG_DFL (LP: #1898026)
  * Protect currently running kernel at run-time (LP: #1615381)
  * Make ADDARG{,C}() macros expand to single statements

 -- Julian Andres Klode <email address hidden> Fri, 12 Mar 2021 09:22:11 +0100

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for apt has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 1.6.13

---------------
apt (1.6.13) bionic; urgency=medium

  [ David Kalnischkies ]
  * Fix incorrect base64 encoding due to int promotion (LP: #1916050)
  * Harden test for no new acquires after transaction abort (Closes: #984966)
    (LP: #1918920)

  [ Julian Andres Klode ]
  * Implement update --error-on=any (Closes: #594813) (LP: #1693900)
  * Include all translations when building the cache (LP: #1907850)
  * Add basic support for the Protected field
  * Do not require force-loopbreak on Important packages
    (Closes: #983014) (LP: #1916725)
  * Protect currently running kernel at run-time (LP: #1615381)
  * Make ADDARG{,C}() macros expand to single statements
  * Improve immediate configuration handling (LP: #1871268)
    - Do not immediately configure m-a: same packages in lockstep
    - Ignore failures from immediate configuration. This does not change the
      actual installation ordering - we never passed the return code to the
      caller and installation went underway anyway if it could be ordered at a
      later stage, this just removes spurious after-the-fact errors.
      (Closes: #973305, #188161, #211075, #649588)
  * Default Acquire::AllowReleaseInfoChange::Suite to "true" (Closes: #931566)
    (LP: #1918907)

  [ Balint Reczey ]
  * Set LC_ALL=C.UTF-8 for unattended-upgrades environment when parsing its --help
    (LP: #1806076)

 -- Julian Andres Klode <email address hidden> Fri, 12 Mar 2021 14:09:15 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 2.0.5

---------------
apt (2.0.5) focal; urgency=medium

  [ Julian Andres Klode ]
  * private-search: Only use V.TranslatedDescription() if good (LP: #1877987)
  * Implement update --error-on=any (Closes: #594813) (LP: #1693900)
  * Include all translations when building the cache (LP: #1907850)
  * Add basic support for the Protected field, and do not require force-loopbreak
    on Protected/Important packages (Closes: #983014) (LP: #1916725)
  * Protect currently running kernel at run-time (LP: #1615381)
  * Make ADDARG{,C}() macros expand to single statements
  * Default Acquire::AllowReleaseInfoChange::Suite to "true" (Closes: #931566)
    (LP: #1918907)

  [ David Kalnischkies ]
  * Fix incorrect base64 encoding due to int promotion (LP: #1916050)
  * Harden test for no new acquires after transaction abort (Closes: #984966)
    (LP: #1918920)

 -- Julian Andres Klode <email address hidden> Fri, 12 Mar 2021 12:47:30 +0100