Please compress packages with zstd by default

Bug #1923845 reported by Balint Reczey on 2021-04-14
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
appstream-glib (Ubuntu)
Undecided
Unassigned
apt (Ubuntu)
High
Unassigned
aptly (Ubuntu)
Undecided
Unassigned
boinc (Ubuntu)
Undecided
Unassigned
busybox (Ubuntu)
Undecided
Unassigned
cdebootstrap (Ubuntu)
Undecided
Unassigned
cdist (Ubuntu)
Undecided
Unassigned
debdelta (Ubuntu)
Undecided
Unassigned
debian-el (Debian)
Unknown
Unknown
debian-el (Ubuntu)
Undecided
Unassigned
debootstrap (Ubuntu)
High
Unassigned
debsig-verify (Ubuntu)
Undecided
Unassigned
debsigs (Ubuntu)
Undecided
Unassigned
diffoscope (Ubuntu)
Wishlist
Unassigned
dpkg (Ubuntu)
Undecided
Unassigned
dpkg-sig (Ubuntu)
Undecided
Unassigned
file (Ubuntu)
Undecided
Unassigned
hello (Ubuntu)
Undecided
Unassigned
libsolv (Ubuntu)
Undecided
Unassigned
lintian (Ubuntu)
Undecided
Unassigned
lutris (Ubuntu)
Undecided
Unassigned
obs-build (Ubuntu)
Undecided
Unassigned
osc (Ubuntu)
Undecided
Unassigned
radare2 (Ubuntu)
Undecided
Unassigned
reprepro (Ubuntu)
Undecided
Unassigned
vim-scripts (Ubuntu)
Undecided
Unassigned
zeroinstall-injector (Ubuntu)
Undecided
Unassigned

Bug Description

https://people.canonical.com/~rbalint/zstd-debs/ contains a .deb built on Hirsute having both data and control members of the .deb being compressed with zstd. It can be handy for testing various tools.

[dpkg]
Decompression support in dpkg landed first in Bionic and is being SRUd to Xenial in LP: #1764220 enable Launchpad's Xenial systems to process the zstd-compressed binary packages.
From dpkg's perspective the upgrade path is cleared.

The original plan was compressing only the internal data.tar .deb member, but dpkg uses uniform compression by default since dpkg 1.19.0 thus I'm collecting all the changes to support control.tar.zst, too, in this bug.

Reviewed packages from:
https://codesearch.debian.net/search?q=data.tar.xz&literal=1&perpkg=1
https://codesearch.debian.net/search?q=control.tar.xz&literal=1&perpkg=1

appstream-glib - needs fix: libappstream-builder/asb-package-deb.c
aptly - needs fix: deb/deb.go
boinc - needs fix: debian/fetch_example_applications.sh
busybox - needs fix: archival/dpkg_deb.c archival/dpkg.c
cdebootstrap - needs fix: src/package.c
cdist - may need fix, can use dpkg-deb: cdist/preos/debootstrap/files/devuan-debootstrap/functions
debdelta - needs fix: debdelta debpatch.sh
debian-el - needs fix: deb-view.el
debian-handbook - needs fix, maybe later, for Debian
debootstrap - needs fix, https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/54
debsigs - needs fix, debsigs
debsig-verify - needs fix, src/debsig-verify.c
diffoscope - needs fix, diffoscope/comparators/deb.py
dpkg - needs fix, change default
dpkg-sig - needs fix, dpkg-sig
dpmb - needs fix, maybe later, for Debian
elfutils - may need fix, uses dpkg-deb if it is available, does not handle .gz either
file - needs fix, magic/Magdir/archive
libsolv - needs fix, ext/repo_deb.c
lintian - needs fix malformed-deb-archive
lutris - needs fix, lutris/util/extract.py
obs-build - needs fix Build/Deb.pm
osc - needs fix osc/util/debquery.py control.tar.zst only
python-apt - needs fix apt_inst.DebFile("glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb").control.extractall()
radare2 - needs fix
reprepro - needs fix, debfile.c
vim-scripts - needs fix debPlugin/autoload/deb.vim
winetricks - needs fix when Debian switches src/winetricks
zeroinstall-injector - needs fix src/zeroinstall/archive.ml

acr - skip, does not _have to_ be fixed, just creates packages, see dist/deb_hand.mak
alien - skip, uses dpkg-deb to extract .deb
ansible - not affected, just test data in dbdata.tar.xz
anthy - not affected, just changelog entry
apt - seems fixed already
ceph - not affected in Ubuntu's version
circlator - not affected, just test data
cowdancer - not affected, just documentation
eccodes - skip, just orig-data.tar.xz
eckit - skip, just ...orig-data.tar.xz
firefox - skip, profdata.tar.xz
firefox-esr - skip, profdata.tar.xz
galculator - skip, just changelog
grads - skip, ...orig-data.tar.xz
gvmd - skip, just creates xz compressed .deb
insighttoolkit4 - skip, ...orig-data.tar.xz
jdeb - skip, just creates compressed .deb packages
jmol - skip, just local data archive
jq - skip, extracting .deb in Tracis CI only
libldm - skip, just local data archive
linux - skip, just local data archive
mate-common - skip, just changelog
megaglest - skip, just local data archive
metview - skip, ...orig-data.tar.xz
mir - skip, .gitignore
multiqc - skip, false hit
naev - skip, local data tarball
nanofilt - skip, local data tarball
python-dendropy - skip, just local data archive
python-gear - skip, just changelog
python-x2go - skip, just changelog
salt - skip, extracts only a fixture
streamlink - skip, changelog only
thunderbird - skip, local data tarballs
velvet - skip, changelog only
veyon - skip, just changelog
victoriametrics - skip, just assembles .deb
wlcs - skip, Travis CI only
wpa - skip, changelog only

Balint Reczey (rbalint) on 2021-04-14
description: updated
Revision history for this message
Dimitri John Ledkov (xnox) wrote :

# file perl-base_5.32.1-3ubuntu2_amd64.deb
perl-base_5.32.1-3ubuntu2_amd64.deb: Debian binary package (format 2.0), with control.tar.zs, data compression zst

Looks odd

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

# display 3rd character of file name extension like 2 of bz2 or m of lzma
>>>>>&-1 ubyte x \b%c
>>>>>>&0 ubyte !0x20
>>>>>>>&-1 ubyte !0x2f

is odd.

Balint Reczey (rbalint) on 2021-04-19
description: updated
Balint Reczey (rbalint) on 2021-04-20
description: updated
Balint Reczey (rbalint) on 2021-04-21
description: updated
Revision history for this message
Balint Reczey (rbalint) wrote :
Changed in python-apt (Ubuntu):
importance: Undecided → High
Changed in debootstrap (Ubuntu):
importance: Undecided → High
status: New → In Progress
Revision history for this message
Julian Andres Klode (juliank) wrote :

I don't understand what python-apt does here, it inherits zstd support from apt.

Revision history for this message
Julian Andres Klode (juliank) wrote :

There seems to be a bug in bionic's apt.

root@bb:/home/jak/Projects# python3 -c 'import apt_inst, apt_pkg; apt_pkg.init(); apt_inst.DebFile("glibc-doc-reference_2.33-0ubuntu2_zstd1_all.deb").control.extractdata("control")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
LookupError: There is no member named 'control'
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 72, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 30, in <module>
    import apport.fileutils
  File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 25, in <module>
    from apport.packaging_impl import impl as packaging
  File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 24, in <module>
    import apt
  File "/usr/lib/python3/dist-packages/apt/__init__.py", line 35, in <module>
    apt_pkg.init_system()
apt_pkg.Error: E:Cannot find a configured compressor for 'false'

no longer affects: python-apt (Ubuntu)
Revision history for this message
Julian Andres Klode (juliank) wrote :

I have not tested hirsute in a clean container; but touch /usr/bin/zstd makes it work, so the compressor lookup is broken.

Revision history for this message
Balint Reczey (rbalint) wrote :

@juliank Yes, python-apt works with just touched /usr/bin/zstd :

>>> apt_inst.DebFile("glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb").control.extractall()
True

Balint Reczey (rbalint) on 2021-04-22
Changed in apt (Ubuntu):
importance: Undecided → High
Revision history for this message
Balint Reczey (rbalint) wrote :

@juliank Xenial's apt/python3-apt does not work even with zstd installed:

root@x-zstd:~# python3
Python 3.5.2 (default, Jan 26 2021, 13:30:48)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import apt_inst, apt_pkg
>>> apt_inst.DebFile("glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb").control.extractall()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
SystemError: E:Internal error, could not locate member control.tar.{lz4gzxzbz2lzma}
>>>
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=====================================-=======================-=======================-================================================================================
ii zstd 1.3.1+dfsg-1~ubuntu0.16 amd64 fast lossless compression algorithm -- CLI tool

Revision history for this message
Julian Andres Klode (juliank) wrote :

Aside from the code being wrong - it's missing an apt_pkg.init_config(), it also does not work with the config being setup; it needs to be whitelisted.

So we definitely need to SRU APT. However, we are already preparing 1.2.35 at the moment for ESM enablement, I don't know if we want to stuff both things together.

Revision history for this message
Balint Reczey (rbalint) wrote :

@juliank IMO it would be better to add the zstd-related changes, too, to the planned SRU if those are not risky and the delay due to adding them is acceptable for ESM's perspective.

Revision history for this message
Julian Andres Klode (juliank) wrote :

Split out APT / python-apt into bug 1923845

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package debootstrap - 1.0.124

---------------
debootstrap (1.0.124) experimental; urgency=medium

  [ Colin Watson ]
  * Add (Ubuntu) groovy as a symlink to gutsy.

  [ Samuel Thibault ]
  * functions: Turn for loops variables into locals, suggested by jhcha54008.

  [ YunQiang Su ]
  * stage1: re-mkdir /proc if it is a symlink instead of umount. (Closes: #968927)

  [ Luca Falavigna ]
  * Add (Ubuntu) hirsute as a symlink to gutsy.

  [ Tobias Koch ]
  * Add --inrelease-path option to allow specifying by-hash InRelease
    location.

  [ Dimitri John Ledkov ]
  * functions: add zstd control&data support to the ar extractor LP: #1923845
  * Add (Ubuntu) impish as a symlink to gutsy.
  * gutsy: sync changes from Ubuntu, to ensure that impish+ are always
    merged-usr.
  * gutsy: use --extractor=ar if dpkg-deb has no zstd support (i.e. on
    Debian & non-Ubuntu derivatives).
  * rules,control: adjust Recommends/Suggests to be appropriate on each
    distro.

 -- Dimitri John Ledkov <email address hidden> Mon, 26 Apr 2021 15:02:44 +0100

Changed in debootstrap (Ubuntu):
status: In Progress → Fix Released
Revision history for this message
Balint Reczey (rbalint) wrote :

Verified 1.6.5ubuntu0.6 on Bionic.

root@bb-lp1926437:~# wget https://people.canonical.com/~rbalint/zstd-debs/glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb
--2021-05-13 19:29:17-- https://people.canonical.com/~rbalint/zstd-debs/glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb
Resolving people.canonical.com (people.canonical.com)... 91.189.89.62
Connecting to people.canonical.com (people.canonical.com)|91.189.89.62|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5618446 (5.4M) [application/x-debian-package]
Saving to: ‘glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb’

glibc-doc-reference_2.33-0ubuntu2~zstd1_all 100%[=========================================================================================>] 5.36M 4.80MB/s in 1.1s

2021-05-13 19:29:18 (4.80 MB/s) - ‘glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb’ saved [5618446/5618446]

root@bb-lp1926437:~# python3
Python 3.6.9 (default, Jan 26 2021, 15:33:00)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import apt_inst
>>> import apt_pkg
>>> apt_pkg.init_config()
>>> apt_inst.DebFile("glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb").control.extractall()
True
>>>
root@bb-lp1926437:~# dpkg -l python3-apt python-apt-common
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=====================================-=======================-=======================-================================================================================
ii python-apt-common 1.6.5ubuntu0.6 all Python interface to libapt-pkg (locales)
ii python3-apt 1.6.5ubuntu0.6 amd64 Python 3 interface to libapt-pkg
root@bb-lp1926437:~#

Revision history for this message
Balint Reczey (rbalint) wrote :

Verified 2.2.0~ubuntu0.21.04.1 on Hirsute.

root@hh-lp1926437:~# wget https://people.canonical.com/~rbalint/zstd-debs/glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb
--2021-05-13 19:42:59-- https://people.canonical.com/~rbalint/zstd-debs/glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb
Resolving people.canonical.com (people.canonical.com)... 91.189.89.62
Connecting to people.canonical.com (people.canonical.com)|91.189.89.62|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5618446 (5.4M) [application/x-debian-package]
Saving to: ‘glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb’

glibc-doc-reference_2.33-0ubuntu2~zstd1_all 100%[=========================================================================================>] 5.36M 4.78MB/s in 1.1s

2021-05-13 19:43:00 (4.78 MB/s) - ‘glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb’ saved [5618446/5618446]

root@hh-lp1926437:~# python3 -c 'import apt_inst; import apt_pkg; apt_pkg.init_config(); print(apt_inst.DebFile("glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb").control.extractall())'
True
root@hh-lp1926437:~# dpkg -l python3-apt python-apt-common
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=================-=====================-============-========================================
ii python-apt-common 2.2.0~ubuntu0.21.04.1 all Python interface to libapt-pkg (locales)
ii python3-apt 2.2.0~ubuntu0.21.04.1 amd64 Python 3 interface to libapt-pkg

Revision history for this message
Balint Reczey (rbalint) wrote :

Verified 2.1.3ubuntu1.4 on Groovy.

root@gg-lp1926437:~# wget https://people.canonical.com/~rbalint/zstd-debs/glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb
--2021-05-13 19:51:34-- https://people.canonical.com/~rbalint/zstd-debs/glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb
Resolving people.canonical.com (people.canonical.com)... 91.189.89.62
Connecting to people.canonical.com (people.canonical.com)|91.189.89.62|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5618446 (5.4M) [application/x-debian-package]
Saving to: ‘glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb’

glibc-doc-reference_2.33-0ubuntu2~zstd1_all 100%[=========================================================================================>] 5.36M 4.70MB/s in 1.1s

2021-05-13 19:51:36 (4.70 MB/s) - ‘glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb’ saved [5618446/5618446]

root@gg-lp1926437:~# python3 -c 'import apt_inst; import apt_pkg; apt_pkg.init_config(); print(apt_inst.DebFile("glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb").control.extractall())'
True
root@gg-lp1926437:~# dpkg -l python3-apt python-apt-common
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=================-==============-============-========================================
ii python-apt-common 2.1.3ubuntu1.4 all Python interface to libapt-pkg (locales)
ii python3-apt 2.1.3ubuntu1.4 amd64 Python 3 interface to libapt-pkg

Revision history for this message
Balint Reczey (rbalint) wrote :

Verified 2.0.0ubuntu0.20.04.5 on Focal.

root@f-lp1926437:~# wget https://people.canonical.com/~rbalint/zstd-debs/glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb
--2021-05-13 19:56:02-- https://people.canonical.com/~rbalint/zstd-debs/glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb
Resolving people.canonical.com (people.canonical.com)... 91.189.89.62
Connecting to people.canonical.com (people.canonical.com)|91.189.89.62|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5618446 (5.4M) [application/x-debian-package]
Saving to: ‘glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb’

glibc-doc-reference_2.33-0ubuntu2~zstd1_all 100%[=========================================================================================>] 5.36M 4.61MB/s in 1.2s

2021-05-13 19:56:03 (4.61 MB/s) - ‘glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb’ saved [5618446/5618446]

root@f-lp1926437:~# python3 -c 'import apt_inst; import apt_pkg; apt_pkg.init_config(); print(apt_inst.DebFile("glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb").control.extractall())'
True
root@f-lp1926437:~# dpkg -l python3-apt python-apt-common
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=================-====================-============-========================================
ii python-apt-common 2.0.0ubuntu0.20.04.5 all Python interface to libapt-pkg (locales)
ii python3-apt 2.0.0ubuntu0.20.04.5 amd64 Python 3 interface to libapt-pkg

Revision history for this message
Balint Reczey (rbalint) wrote :

Sorry, those comments were for LP: #1926437, which is at least related. :-)

Changed in apt (Ubuntu):
status: New → Fix Released
Revision history for this message
Balint Reczey (rbalint) wrote :

The attached dpkg patch has been tested in the following Bileto ticket:

https://bileto.ubuntu.com/#/ticket/4583

Lintian needs to be fixed first to ensure smooth migration. The software-properties failure does not seem to be related at first glance.

Revision history for this message
Balint Reczey (rbalint) wrote :
Revision history for this message
Dimitri John Ledkov (xnox) wrote :

dpkg patch looks good to me.

did lintian patch locally and started to run the regression test suite locally to see if anything else is tripped up.

working on reprepro changes too.

tags: added: patch
Revision history for this message
Dimitri John Ledkov (xnox) wrote :

lintian should be good in impish-proposed, but we should migrate it first.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package lintian - 2.104.0ubuntu3

---------------
lintian (2.104.0ubuntu3) impish; urgency=medium

  * Add zst as allowed compression for .deb & .udeb. Although we don't
    actually ship udeb package that supports zst. LP: #1923845

 -- Dimitri John Ledkov <email address hidden> Tue, 15 Jun 2021 10:42:12 +0100

Changed in lintian (Ubuntu):
status: New → Fix Released
Revision history for this message
Balint Reczey (rbalint) wrote :

@xnox Thanks! Please also SRU the reprepro changes. Ideally users of reprepro on stable releases would be able to just upgrade reprepro from -updates before the first zstd-compressed .deb lands in Impish to not break their setup even when they maintain a local repository of the devel series.

tags: added: block-proposed-impish
tags: removed: block-proposed-impish
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package hello - 2.10-2ubuntu3

---------------
hello (2.10-2ubuntu3) impish; urgency=medium

  * Ubuntu Impish will be switching default dpkg compression format to
    Zstd. This upload is the first canary to signal the archive format
    change. dpkg in Ubuntu supports zstd in xenial and up. If any issues
    are identified with this package, please report them at
    https://pad.lv/1923845 LP: #1923845

 -- Dimitri John Ledkov <email address hidden> Fri, 18 Jun 2021 11:58:54 +0100

Changed in hello (Ubuntu):
status: New → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package reprepro - 5.3.0-1.3

---------------
reprepro (5.3.0-1.3) experimental; urgency=medium

  * Non-maintainer upload.
  * Add support for Zstd compressed debs. (LP: #1923845)

 -- Dimitri John Ledkov <email address hidden> Mon, 21 Jun 2021 10:16:53 +0100

Changed in reprepro (Ubuntu):
status: New → Fix Released
Revision history for this message
Balint Reczey (rbalint) wrote :
Revision history for this message
Balint Reczey (rbalint) wrote :

Diffoscope already handles zstd compressed members well.

Changed in diffoscope (Ubuntu):
importance: Undecided → Wishlist
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.