[SRU] Backport JSON hooks 0.2

[Julian Andres Klode]

[Impact]
We want to be able to use JSON hooks 0.2 in xenial and later releases to display messages to users during dist-upgrade.

For xenial, we backport the JSON hook support in its entirety, whereas for later releases, we only need to cherry-pick bugfixes and the v0.2 improvements.

[Test plan]
We have included unit and integration tests for the new JSON code to test the writer and prevent regressions in future changes. Regressions in other parts of the code should be caught by the test suite.

[Where problems could occur]
There may be bugs in the JSON code (apt-private/private-json-hooks.*) and in the integration thereof. The JSON hook code is identical to 2.3.2, whereas the integration code needed slight adjustments.

The integration changes are limited to adding the hooks, and (for xenial, already in later releases) collecting all unknown package names. We believe the regression potential of these changes to be fairly limited.

However, errors in JSON hooks are fatal, so once JSON hooks are running, they can cause apt to fail, which has to be kept in mind.

Existing JSON hooks might not expect the new notifications, and break, but we are only aware of the snapd hook and it ignores any unknown notifications, so that should not pose a problem.

[Julian Andres Klode (juliank)]

These changes are in apt 2.3.2 which is queued for impish.

[Brian Murray (brian-murray)]

Hello Julian, or anyone else affected,

Accepted apt into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/1.2.35 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted apt (1.2.35) for xenial have finished running.
The following regressions have been reported in tests triggered by the package:

apt/1.2.35 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#apt

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 1.2.35

---------------
apt (1.2.35) xenial; urgency=medium

  * Backport JSON hooks, version 0.2, to xenial (LP: #1926150). The JSON code
    files are identical to that of 2.3.2, only the integration and test cases
    needed minor adjustment to behave correctly, especially:
    - In private-install.cc, exit before showing the list of packages to
      upgrade/install/etc, in case an error is already set. This moves the
      behavior closer to bionic.
  * Backport zstd support for Launchpad zstd enablement (LP: #1926437)
  * Fix indendation of changelog message in 1.2.34 changelog.
  * Bug fixes needed for JSON hooks:
    - private-install: Handle existing errors before showing lists
    - Avoid duplicated error in `apt search`
  * Bug fixes affecting CI / autopkgtest only:
    - prepare-release: Ignore alternative build dependencies
    - tests: Do not expect requested-by if sudo was invoked by root
    - tests: Export TZ=UTC to work around test failures on non-UTC hosts
    - tests: avoid time-dependent rebuild of caches

 -- Julian Andres Klode <email address hidden> Wed, 28 Apr 2021 14:55:54 +0200

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for apt has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Brian Murray (brian-murray)]

apt (2.3.2) experimental; urgency=medium

  * Remove inversed comment for AllowUnsizedPackages
  * Automatically retry failed downloads 3 times.
    Enable the Acquire::Retries option by default, set to 3.
    This will help with slightly unreliable networking; future
    work is needed for adding backoff and SRV/IP rotation. (LP: #1876035)
  * Require dpkg >= 1.20.8 and make more use of --auto-deconfigure now that
    no longer needs forcing for essential/protected:
    - Support deconfiguring Essential packages
    - Do not pass --force-remove-protected with --auto-deconfigure
  * JSON hook bug fixes:
    - json: Escape strings using \u escape sequences, add test
    - json: Actually pop states
    - json: Encode NULL strings as null
    - json: Flush standard file descriptors before calling hooks
    - test/json: Make the test hook more reliable
  * JSON hooks 0.2:
    - json: Add origins fields to version
    - upgrade: Add JSON hook support (AptCli::Hooks::Upgrade)
    - json: Add `package-list` and `statistics` install hooks
    - json: Hook protocol 0.2 (added upgrade,downgrade,reinstall modes)
    - 2.3-only: Warn that the 0.1 protocol is deprecated
  * Avoid infinite loop on EOF on media change prompt

I think this is fixed for Impish.

[Brian Murray (brian-murray)]

Hello Julian, or anyone else affected,

Accepted apt into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/2.2.4ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Julian, or anyone else affected,

Accepted apt into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/2.0.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Julian, or anyone else affected,

Accepted apt into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/1.6.14 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted apt (1.6.14) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

apport/2.20.9-0ubuntu7.24 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#apt

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted apt (2.0.6) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

reprotest/0.7.14 (ppc64el)
apport/2.20.11-0ubuntu27.18 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#apt

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted apt (2.2.4ubuntu0.1) for hirsute have finished running.
The following regressions have been reported in tests triggered by the package:

ubuntu-advantage-tools/unknown (amd64, ppc64el, s390x, armhf)
reprotest/0.7.16 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/hirsute/update_excuses.html#apt

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Julian Andres Klode (juliank)]

autopkgtests for apt itself have passed, verifying the fix. We still battle with one apport regression on focal, but it seems unrelated - they've failed before.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 2.0.6

---------------
apt (2.0.6) focal; urgency=medium

  * RunScripts: Do not reset SIGQUIT and SIGINT to SIG_DFL (LP: #1898026)
  * test suite fix: Check for and discard expected warning from MaybeAddAuth
  * Fix downloads of unsized files that are largest in pipeline (LP: #1921626),
    and warn about packages without size (option Acquire::AllowUnsizedPackages)
  * JSON hooks 0.2 and assorted JSON bugfixes (LP: #1926150)
    - encoder fixes:
      + json: Escape strings using \u escape sequences, add test
      + json: Actually pop states
      + json: Encode NULL strings as null
    - json: Flush standard file descriptors before calling hooks
      (this avoids output from hooks in middle of apt output)
    - non-code changes:
      + test/json: Make the test hook more reliable
      + Fix a typo in json-hooks-protocol.md (thanks to Brian Murray)
    - semantic changes (new fields, hooks, and protocol 0.2):
      + json: Add origins fields to version
      + upgrade: Add JSON hook support (AptCli::Hooks::Upgrade)
      + json: Add `package-list` and `statistics` install hooks
      + json: Hook protocol 0.2 (added upgrade,downgrade,reinstall modes)
    + Fix a typo in json-hooks-protocol.md (thanks to Brian Murray)
  * Avoid infinite loop on EOF on media change prompt (LP: #1928687)

 -- Julian Andres Klode <email address hidden> Tue, 15 Jun 2021 13:05:46 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 1.6.14

---------------
apt (1.6.14) bionic; urgency=medium

  * RunScripts: Do not reset SIGQUIT and SIGINT to SIG_DFL (LP: #1898026)
  * Fix downloads of unsized files that are largest in pipeline (LP: #1921626),
    and warn about packages without size (option Acquire::AllowUnsizedPackages)
  * JSON hooks 0.2 and assorted JSON bugfixes (LP: #1926150)
    - encoder fixes:
      + json: Escape strings using \u escape sequences, add test
      + json: Actually pop states
      + json: Encode NULL strings as null
    - json: Flush standard file descriptors before calling hooks
      (this avoids output from hooks in middle of apt output)
    - Minor fixes to include and C++ namespaces
    - non-code changes:
      + test/json: Make the test hook more reliable
      + Fix a typo in json-hooks-protocol.md (thanks to Brian Murray)
    - semantic changes (new fields, hooks, and protocol 0.2):
      + json: Add origins fields to version
      + upgrade: Add JSON hook support (AptCli::Hooks::Upgrade)
      + json: Add `package-list` and `statistics` install hooks
      + json: Hook protocol 0.2 (added upgrade,downgrade,reinstall modes)
    + Fix a typo in json-hooks-protocol.md (thanks to Brian Murray)
  * Avoid infinite loop on EOF on media change prompt (LP: #1928687)

 -- Julian Andres Klode <email address hidden> Tue, 15 Jun 2021 16:12:38 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 2.2.4ubuntu0.1

---------------
apt (2.2.4ubuntu0.1) hirsute; urgency=medium

  * Merge 2.2.4 from Debian unstable
  * Extend the JSON hook fixes with support for version 0.2 (LP: #1926150)
    - json: Add origins fields to version
    - upgrade: Add JSON hook support (AptCli::Hooks::Upgrade)
    - json: Add `package-list` and `statistics` install hooks
    - json: Hook protocol 0.2 (added upgrade,downgrade,reinstall modes)

apt (2.2.4) unstable; urgency=medium

  [ Julian Andres Klode ]
  * Various bugfixes to the JSON hooks:
    - encoder fixes:
      + json: Escape strings using \u escape sequences, add test
      + json: Actually pop states
      + json: Encode NULL strings as null
    - json: Flush standard file descriptors before calling hooks
      (this avoids output from hooks in middle of apt output)
    - Non-installed JSON changes:
      + test/json: Make the test hook more reliable
      + Fix a typo in json-hooks-protocol.md (thanks to Brian Murray)
  * Avoid infinite loop on EOF on media change prompt (LP: #1928687)
  * Turn TLS handshake issues into transient errors (LP: #1928100),
    this makes behavior consistent with TCP and enables Acquire::Retries
  * policy: Apply phasing to uninstalled packages too (LP: #1925745),
    this prevents inconsistencies when installing new binaries that depend
    on the same version of an already installed binary.

  [ David Kalnischkies ]
  * URI encode Filename field of Packages files (again). This fixes a
    regression introduced in 2.1.15 that causes download failures of
    packages with an epoch included in their filename (LP: #1931874).

 -- Julian Andres Klode <email address hidden> Mon, 14 Jun 2021 16:27:40 +0200