[SRU] Backport zstd support, fix bug in python-apt

[Julian Andres Klode]

[Impact]
APT in xenial needs to learn about zstd support to enable Launchpad to work with zstd packages.

python-apt in all releases needs to be adjusted to pass the compressor names instead of programs to ExtractTar, as otherwise, if /usr/bin/zstd does not exist, it will pass "false" as the compressor, and then ExtractTar fails (same for other compressors).

On hirsute, the python-apt upload also includes some new <!nocheck> annotations for build-dependencies, as it's a straight upload of the 2.2
stable series.

[Test plan]
The apt regression tests cover the compression methods extensively, including even stored compressed lists in /var/lib/apt and fetching Packages files compressed with such methods, for all compression methods.

The final test is to make apt_inst.DebFile("glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb").control.extractall() from 1923845 work in python3-apt.

For hirsute, the build dependency changes will verify themselves during the build ;)

[Where problems could occur]

APT:

The changes to APT are adding the zstd compression backend to FileFd and then linking it into the configuration. The former is adding new code, triggered only on zstd files, and should not be able to cause regressions.

The latter is a bunch of small lines here and there, and it caused issues during backporting in the test suite because it expected compressors in a specific order. However, chances are that nothing else has as stringent requirements as the test suite, so I don't believe that is a problem for existing users.

Bugs can of course be in the code dealing with zstd, and that might open you up to security vulnerabilities in the zstd code if you pass APT zstd compressed files.

A regression can be if clients used apt-helper cat-file to cat a zstd-compressed file ending in .zst - they'll now get uncompressed content, rather than compressed.

python-apt:

The change just replaces Binary with Name, aligning the code with APT itself, so it's so trivial, we don't expect regressions.

[Brian Murray (brian-murray)]

Hello Julian, or anyone else affected,

Accepted apt into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/1.2.35 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted apt (1.2.35) for xenial have finished running.
The following regressions have been reported in tests triggered by the package:

apt/1.2.35 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#apt

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Julian Andres Klode (juliank)]

autopkgtests for apt have passed. I've build a patched version of python-apt and verified that it worked with the apt version, so the apt side is verified now :D

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 1.2.35

---------------
apt (1.2.35) xenial; urgency=medium

  * Backport JSON hooks, version 0.2, to xenial (LP: #1926150). The JSON code
    files are identical to that of 2.3.2, only the integration and test cases
    needed minor adjustment to behave correctly, especially:
    - In private-install.cc, exit before showing the list of packages to
      upgrade/install/etc, in case an error is already set. This moves the
      behavior closer to bionic.
  * Backport zstd support for Launchpad zstd enablement (LP: #1926437)
  * Fix indendation of changelog message in 1.2.34 changelog.
  * Bug fixes needed for JSON hooks:
    - private-install: Handle existing errors before showing lists
    - Avoid duplicated error in `apt search`
  * Bug fixes affecting CI / autopkgtest only:
    - prepare-release: Ignore alternative build dependencies
    - tests: Do not expect requested-by if sudo was invoked by root
    - tests: Export TZ=UTC to work around test failures on non-UTC hosts
    - tests: avoid time-dependent rebuild of caches

 -- Julian Andres Klode <email address hidden> Wed, 28 Apr 2021 14:55:54 +0200

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for apt has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Julian Andres Klode (juliank)]

FWIW, I upgraded python-apt to the version in xenial-proposed

Unpacking python3-apt (1.1.0~beta1ubuntu0.16.04.12) over (1.1.0~beta1ubuntu0.16.04.11) ...

And verified that the script

import apt_inst, apt_pkg
apt_pkg.init()
apt_inst.DebFile("glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb").control.extractall()

worked.

So verification-done-xenial is valid for this one too :D

apport fails weirdly, not have many other autopkgtest results back yet.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted python-apt (1.1.0~beta1ubuntu0.16.04.12) for xenial have finished running.
The following regressions have been reported in tests triggered by the package:

botch/unknown (arm64)
update-notifier/unknown (arm64)
snapcraft/unknown (arm64)
ubiquity/unknown (arm64)
isenkram/unknown (arm64)
python-apt/unknown (arm64)
update-manager/unknown (arm64)
apport/2.20.1-0ubuntu2.30 (amd64, i386)
snapcraft/2.43.1+16.04.1 (amd64, armhf, i386)
bzr-builddeb/unknown (arm64)
firefox/unknown (arm64)
chromium-browser/unknown (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#python-apt

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Łukasz Zemczak (sil2100)]

Weirdly there was no acceptance comment regarding python-apt for xenial, but I confirm that the package has been accepted into xenial-proposed 10 days ago. Let's adjust the status.

Julian, what about the failing autopkgtests for python-apt? Did you have a moment to take a look at those?

[Łukasz Zemczak (sil2100)]

Hello Julian, or anyone else affected,

Accepted python-apt into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-apt/2.2.0~ubuntu0.21.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Julian, or anyone else affected,

Accepted python-apt into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-apt/2.1.3ubuntu1.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Julian, or anyone else affected,

Accepted python-apt into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-apt/2.0.0ubuntu0.20.04.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Julian, or anyone else affected,

Accepted python-apt into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-apt/1.6.5ubuntu0.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted python-apt (2.2.0~ubuntu0.21.04.1) for hirsute have finished running.
The following regressions have been reported in tests triggered by the package:

apport/2.20.11-0ubuntu65 (amd64)
sshuttle/1.0.4-1ubuntu4 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/hirsute/update_excuses.html#python-apt

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted python-apt (2.0.0ubuntu0.20.04.5) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

unattended-upgrades/2.3ubuntu0.1 (armhf)
apt-clone/0.4.1ubuntu3 (ppc64el, amd64, arm64, s390x, armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#python-apt

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted python-apt (2.1.3ubuntu1.4) for groovy have finished running.
The following regressions have been reported in tests triggered by the package:

apt-clone/0.4.1ubuntu5 (armhf, ppc64el, amd64, s390x, arm64)
apt/2.1.10ubuntu0.3 (s390x)
salt/3001+dfsg1-2 (s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/groovy/update_excuses.html#python-apt

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted python-apt (1.6.5ubuntu0.6) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

botch/unknown (arm64)
update-manager/unknown (arm64)
apt/1.6.13 (amd64)
ubiquity/unknown (arm64)
isenkram/unknown (arm64)
python-apt/unknown (arm64)
chromium-browser/unknown (arm64)
livecd-rootfs/unknown (arm64)
aptdaemon/1.1.1+bzr982-0ubuntu19.5 (i386)
breezy-debian/unknown (arm64)
bzr-builddeb/unknown (arm64)
apport/2.20.9-0ubuntu7.23 (amd64)
software-properties/0.96.24.32.14 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#python-apt

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Julian Andres Klode (juliank)]

The xenial SRU should be good to go now, the snapcraft failures are unrelated and also reproduced in vorlon's retry of snapcraft itself.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-apt - 2.2.0

---------------
python-apt (2.2.0) unstable; urgency=medium

  [ Helmut Grohne ]
  * Annotate test dependencies with <!nocheck> (Closes: #980719)

  [ Julian Andres Klode ]
  * debian/rules: Re-enable PIE hardening
  * Mirror list updates
  * debfile: Pass `Name` instead of `Binary` to ExtractTar.
    Passing the binary causes it to fail trying to find the `false`
    compressor when a binary for a given compressor is not installed.
    (LP: #1926437)
  * debian/gbp.conf: Branch name is main

 -- Julian Andres Klode <email address hidden> Thu, 29 Apr 2021 09:49:08 +0200

[Balint Reczey (rbalint)]

Verified 1.6.5ubuntu0.6 on Bionic.

root@bb-lp1926437:~# wget https://people.canonical.com/~rbalint/zstd-debs/glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb
--2021-05-13 19:29:17-- https://people.canonical.com/~rbalint/zstd-debs/glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb
Resolving people.canonical.com (people.canonical.com)... 91.189.89.62
Connecting to people.canonical.com (people.canonical.com)|91.189.89.62|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5618446 (5.4M) [application/x-debian-package]
Saving to: ‘glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb’

glibc-doc-reference_2.33-0ubuntu2~zstd1_all 100%[=========================================================================================>] 5.36M 4.80MB/s in 1.1s

2021-05-13 19:29:18 (4.80 MB/s) - ‘glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb’ saved [5618446/5618446]

root@bb-lp1926437:~# python3
Python 3.6.9 (default, Jan 26 2021, 15:33:00)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import apt_inst
>>> import apt_pkg
>>> apt_pkg.init_config()
>>> apt_inst.DebFile("glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb").control.extractall()
True
>>>
root@bb-lp1926437:~# dpkg -l python3-apt python-apt-common
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=====================================-=======================-=======================-================================================================================
ii python-apt-common 1.6.5ubuntu0.6 all Python interface to libapt-pkg (locales)
ii python3-apt 1.6.5ubuntu0.6 amd64 Python 3 interface to libapt-pkg
root@bb-lp1926437:~#

Verified 2.0.0ubuntu0.20.04.5 on Focal.

root@f-lp1926437:~# wget https://people.canonical.com/~rbalint/zstd-debs/glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb
--2021-05-13 19:56:02-- https://people.canonical.com/~rbalint/zstd-debs/glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb
Resolving people.canonical.com (people.canonical.com)... 91.189.89.62
Connecting to people.canonical.com (people.canonical.com)|91.189.89.62|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5618446 (5.4M) [application/x-debian-package]
Saving to: ‘glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb’

glibc-doc-reference_2.33-0ubuntu2~zstd1_all 100%[=========================================================================================>] 5.36M 4.61MB/s in 1.2s

2021-05-13 19:56:03 (4.61 MB/s) - ‘glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb’ saved [5618446/5618446]

root@f-lp1926437:~# python3 -c 'import apt_inst; import apt_pkg; apt_pkg.init_config(); print(apt_inst.DebFile("glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb").control.extractall())'
True
root@f-lp1926437:~# dpkg -l python3-apt python-apt-common
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==========...

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-apt - 2.2.0~ubuntu0.21.04.1

---------------
python-apt (2.2.0~ubuntu0.21.04.1) hirsute; urgency=medium

  * Upload to hirsute; changes since 2.1.7ubuntu2:
    - Annotate test dependencies with <!nocheck> (Closes: #980719)
    - Mirror list updates
    - debfile: Pass `Name` instead of `Binary` to ExtractTar.
      Passing the binary causes it to fail trying to find the `false`
      compressor when a binary for a given compressor is not installed.
      (LP: #1926437)
    - debian/gbp.conf: Branch name is main

python-apt (2.2.0) unstable; urgency=medium

  [ Helmut Grohne ]
  * Annotate test dependencies with <!nocheck> (Closes: #980719)

  [ Julian Andres Klode ]
  * debian/rules: Re-enable PIE hardening
  * Mirror list updates
  * debfile: Pass `Name` instead of `Binary` to ExtractTar.
    Passing the binary causes it to fail trying to find the `false`
    compressor when a binary for a given compressor is not installed.
    (LP: #1926437)
  * debian/gbp.conf: Branch name is main

 -- Julian Andres Klode <email address hidden> Thu, 29 Apr 2021 15:20:13 +0200

[Chris Halse Rogers (raof)]

The focal and groovy autopkgtest failures in apt-clone look suspiciously like real regressions. Do we have an analysis of those?

[Brian Murray (brian-murray)]

The apt-clone autopkgtest failures are explained in https://bugs.launchpad.net/ubuntu/+source/apt-clone/+bug/1902830.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-apt - 1.6.5ubuntu0.6

---------------
python-apt (1.6.5ubuntu0.6) bionic; urgency=medium

  * debfile: Pass `Name` instead of `Binary` to ExtractTar.
    Passing the binary causes it to fail trying to find the `false`
    compressor when a binary for a given compressor is not installed.
    (LP: #1926437)
  * Update mirror lists

 -- Julian Andres Klode <email address hidden> Thu, 29 Apr 2021 21:33:02 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-apt - 1.1.0~beta1ubuntu0.16.04.12

---------------
python-apt (1.1.0~beta1ubuntu0.16.04.12) xenial; urgency=medium

  * debfile: Pass `Name` instead of `Binary` to ExtractTar.
    Passing the binary causes it to fail trying to find the `false`
    compressor when a binary for a given compressor is not installed.
    (LP: #1926437)
  * Update mirror lists

 -- Julian Andres Klode <email address hidden> Thu, 29 Apr 2021 21:37:32 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-apt - 2.0.0ubuntu0.20.04.5

---------------
python-apt (2.0.0ubuntu0.20.04.5) focal; urgency=medium

  * debfile: Pass `Name` instead of `Binary` to ExtractTar.
    Passing the binary causes it to fail trying to find the `false`
    compressor when a binary for a given compressor is not installed.
    (LP: #1926437)
  * Update mirror lists

 -- Julian Andres Klode <email address hidden> Thu, 29 Apr 2021 16:12:21 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-apt - 2.1.3ubuntu1.4

---------------
python-apt (2.1.3ubuntu1.4) groovy; urgency=medium

  * debfile: Pass `Name` instead of `Binary` to ExtractTar.
    Passing the binary causes it to fail trying to find the `false`
    compressor when a binary for a given compressor is not installed.
    (LP: #1926437)
  * Update mirror lists

 -- Julian Andres Klode <email address hidden> Thu, 29 Apr 2021 15:31:59 +0200