[SRU] Backport JSON hooks 0.2

Bug #1926150 reported by Julian Andres Klode
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
apt (Ubuntu)
Undecided
Unassigned
Xenial
Undecided
Unassigned
Bionic
Undecided
Unassigned
Focal
Undecided
Unassigned
Hirsute
Undecided
Unassigned
Impish
Undecided
Unassigned

Bug Description

[Impact]
We want to be able to use JSON hooks 0.2 in xenial and later releases to display messages to users during dist-upgrade.

For xenial, we backport the JSON hook support in its entirety, whereas for later releases, we only need to cherry-pick bugfixes and the v0.2 improvements.

[Test plan]
We have included unit and integration tests for the new JSON code to test the writer and prevent regressions in future changes. Regressions in other parts of the code should be caught by the test suite.

[Where problems could occur]
There may be bugs in the JSON code (apt-private/private-json-hooks.*) and in the integration thereof. The JSON hook code is identical to 2.3.2, whereas the integration code needed slight adjustments.

The integration changes are limited to adding the hooks, and (for xenial, already in later releases) collecting all unknown package names. We believe the regression potential of these changes to be fairly limited.

However, errors in JSON hooks are fatal, so once JSON hooks are running, they can cause apt to fail, which has to be kept in mind.

Existing JSON hooks might not expect the new notifications, and break, but we are only aware of the snapd hook and it ignores any unknown notifications, so that should not pose a problem.

Revision history for this message
Julian Andres Klode (juliank) wrote :

These changes are in apt 2.3.2 which is queued for impish.

Changed in apt (Ubuntu Impish):
status: New → Fix Committed
Changed in apt (Ubuntu Xenial):
status: New → Triaged
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Julian, or anyone else affected,

Accepted apt into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/1.2.35 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in apt (Ubuntu Xenial):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-xenial
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (apt/1.2.35)

All autopkgtests for the newly accepted apt (1.2.35) for xenial have finished running.
The following regressions have been reported in tests triggered by the package:

apt/1.2.35 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#apt

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

tags: added: verification-done verification-done-xenial
removed: verification-needed verification-needed-xenial
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apt - 1.2.35

---------------
apt (1.2.35) xenial; urgency=medium

  * Backport JSON hooks, version 0.2, to xenial (LP: #1926150). The JSON code
    files are identical to that of 2.3.2, only the integration and test cases
    needed minor adjustment to behave correctly, especially:
    - In private-install.cc, exit before showing the list of packages to
      upgrade/install/etc, in case an error is already set. This moves the
      behavior closer to bionic.
  * Backport zstd support for Launchpad zstd enablement (LP: #1926437)
  * Fix indendation of changelog message in 1.2.34 changelog.
  * Bug fixes needed for JSON hooks:
    - private-install: Handle existing errors before showing lists
    - Avoid duplicated error in `apt search`
  * Bug fixes affecting CI / autopkgtest only:
    - prepare-release: Ignore alternative build dependencies
    - tests: Do not expect requested-by if sudo was invoked by root
    - tests: Export TZ=UTC to work around test failures on non-UTC hosts
    - tests: avoid time-dependent rebuild of caches

 -- Julian Andres Klode <email address hidden> Wed, 28 Apr 2021 14:55:54 +0200

Changed in apt (Ubuntu Xenial):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for apt has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Brian Murray (brian-murray) wrote :

apt (2.3.2) experimental; urgency=medium

  * Remove inversed comment for AllowUnsizedPackages
  * Automatically retry failed downloads 3 times.
    Enable the Acquire::Retries option by default, set to 3.
    This will help with slightly unreliable networking; future
    work is needed for adding backoff and SRV/IP rotation. (LP: #1876035)
  * Require dpkg >= 1.20.8 and make more use of --auto-deconfigure now that
    no longer needs forcing for essential/protected:
    - Support deconfiguring Essential packages
    - Do not pass --force-remove-protected with --auto-deconfigure
  * JSON hook bug fixes:
    - json: Escape strings using \u escape sequences, add test
    - json: Actually pop states
    - json: Encode NULL strings as null
    - json: Flush standard file descriptors before calling hooks
    - test/json: Make the test hook more reliable
  * JSON hooks 0.2:
    - json: Add origins fields to version
    - upgrade: Add JSON hook support (AptCli::Hooks::Upgrade)
    - json: Add `package-list` and `statistics` install hooks
    - json: Hook protocol 0.2 (added upgrade,downgrade,reinstall modes)
    - 2.3-only: Warn that the 0.1 protocol is deprecated
  * Avoid infinite loop on EOF on media change prompt

I think this is fixed for Impish.

Changed in apt (Ubuntu Impish):
status: Fix Committed → Fix Released
tags: added: fr-1370
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Julian, or anyone else affected,

Accepted apt into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/2.2.4ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in apt (Ubuntu Hirsute):
status: New → Fix Committed
tags: added: verification-needed verification-needed-hirsute
removed: verification-done
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Julian, or anyone else affected,

Accepted apt into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/2.0.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in apt (Ubuntu Focal):
status: New → Fix Committed
tags: added: verification-needed-focal
Changed in apt (Ubuntu Bionic):
status: New → Fix Committed
tags: added: verification-needed-bionic
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Julian, or anyone else affected,

Accepted apt into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/1.6.14 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (apt/1.6.14)

All autopkgtests for the newly accepted apt (1.6.14) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

apport/2.20.9-0ubuntu7.24 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#apt

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (apt/2.0.6)

All autopkgtests for the newly accepted apt (2.0.6) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

reprotest/0.7.14 (ppc64el)
apport/2.20.11-0ubuntu27.18 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#apt

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (apt/2.2.4ubuntu0.1)

All autopkgtests for the newly accepted apt (2.2.4ubuntu0.1) for hirsute have finished running.
The following regressions have been reported in tests triggered by the package:

ubuntu-advantage-tools/unknown (amd64, ppc64el, s390x, armhf)
reprotest/0.7.16 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/hirsute/update_excuses.html#apt

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Julian Andres Klode (juliank) wrote :

autopkgtests for apt itself have passed, verifying the fix. We still battle with one apport regression on focal, but it seems unrelated - they've failed before.

tags: added: verification-done verification-done-bionic verification-done-focal verification-done-hirsute
removed: verification-needed verification-needed-bionic verification-needed-focal verification-needed-hirsute
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apt - 2.0.6

---------------
apt (2.0.6) focal; urgency=medium

  * RunScripts: Do not reset SIGQUIT and SIGINT to SIG_DFL (LP: #1898026)
  * test suite fix: Check for and discard expected warning from MaybeAddAuth
  * Fix downloads of unsized files that are largest in pipeline (LP: #1921626),
    and warn about packages without size (option Acquire::AllowUnsizedPackages)
  * JSON hooks 0.2 and assorted JSON bugfixes (LP: #1926150)
    - encoder fixes:
      + json: Escape strings using \u escape sequences, add test
      + json: Actually pop states
      + json: Encode NULL strings as null
    - json: Flush standard file descriptors before calling hooks
      (this avoids output from hooks in middle of apt output)
    - non-code changes:
      + test/json: Make the test hook more reliable
      + Fix a typo in json-hooks-protocol.md (thanks to Brian Murray)
    - semantic changes (new fields, hooks, and protocol 0.2):
      + json: Add origins fields to version
      + upgrade: Add JSON hook support (AptCli::Hooks::Upgrade)
      + json: Add `package-list` and `statistics` install hooks
      + json: Hook protocol 0.2 (added upgrade,downgrade,reinstall modes)
    + Fix a typo in json-hooks-protocol.md (thanks to Brian Murray)
  * Avoid infinite loop on EOF on media change prompt (LP: #1928687)

 -- Julian Andres Klode <email address hidden> Tue, 15 Jun 2021 13:05:46 +0200

Changed in apt (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apt - 1.6.14

---------------
apt (1.6.14) bionic; urgency=medium

  * RunScripts: Do not reset SIGQUIT and SIGINT to SIG_DFL (LP: #1898026)
  * Fix downloads of unsized files that are largest in pipeline (LP: #1921626),
    and warn about packages without size (option Acquire::AllowUnsizedPackages)
  * JSON hooks 0.2 and assorted JSON bugfixes (LP: #1926150)
    - encoder fixes:
      + json: Escape strings using \u escape sequences, add test
      + json: Actually pop states
      + json: Encode NULL strings as null
    - json: Flush standard file descriptors before calling hooks
      (this avoids output from hooks in middle of apt output)
    - Minor fixes to include and C++ namespaces
    - non-code changes:
      + test/json: Make the test hook more reliable
      + Fix a typo in json-hooks-protocol.md (thanks to Brian Murray)
    - semantic changes (new fields, hooks, and protocol 0.2):
      + json: Add origins fields to version
      + upgrade: Add JSON hook support (AptCli::Hooks::Upgrade)
      + json: Add `package-list` and `statistics` install hooks
      + json: Hook protocol 0.2 (added upgrade,downgrade,reinstall modes)
    + Fix a typo in json-hooks-protocol.md (thanks to Brian Murray)
  * Avoid infinite loop on EOF on media change prompt (LP: #1928687)

 -- Julian Andres Klode <email address hidden> Tue, 15 Jun 2021 16:12:38 +0200

Changed in apt (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apt - 2.2.4ubuntu0.1

---------------
apt (2.2.4ubuntu0.1) hirsute; urgency=medium

  * Merge 2.2.4 from Debian unstable
  * Extend the JSON hook fixes with support for version 0.2 (LP: #1926150)
    - json: Add origins fields to version
    - upgrade: Add JSON hook support (AptCli::Hooks::Upgrade)
    - json: Add `package-list` and `statistics` install hooks
    - json: Hook protocol 0.2 (added upgrade,downgrade,reinstall modes)

apt (2.2.4) unstable; urgency=medium

  [ Julian Andres Klode ]
  * Various bugfixes to the JSON hooks:
    - encoder fixes:
      + json: Escape strings using \u escape sequences, add test
      + json: Actually pop states
      + json: Encode NULL strings as null
    - json: Flush standard file descriptors before calling hooks
      (this avoids output from hooks in middle of apt output)
    - Non-installed JSON changes:
      + test/json: Make the test hook more reliable
      + Fix a typo in json-hooks-protocol.md (thanks to Brian Murray)
  * Avoid infinite loop on EOF on media change prompt (LP: #1928687)
  * Turn TLS handshake issues into transient errors (LP: #1928100),
    this makes behavior consistent with TCP and enables Acquire::Retries
  * policy: Apply phasing to uninstalled packages too (LP: #1925745),
    this prevents inconsistencies when installing new binaries that depend
    on the same version of an already installed binary.

  [ David Kalnischkies ]
  * URI encode Filename field of Packages files (again). This fixes a
    regression introduced in 2.1.15 that causes download failures of
    packages with an epoch included in their filename (LP: #1931874).

 -- Julian Andres Klode <email address hidden> Mon, 14 Jun 2021 16:27:40 +0200

Changed in apt (Ubuntu Hirsute):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers