autopkgtest-satdep.deb unpack fails on Debian images

[Athos Ribeiro]

[Impact]

dpkg defaults to the zstd compression format for compressing the members in a deb package archive. This compression format is not supported in Debian. Therefore, running autopkgtest from an Ubuntu (21.10) host on a Debian target fails, since autopkgtest uses dpkg-deb to generate the autopkgtest-satdep.deb package.

[Test Plan]

For any package with dep8 tests, run

$ autopkgtest-build-lxd images:debian/sid
$ autopkgtest ./. -- lxd autopkgtest/debian/sid/amd64

and verify it fails with:

autopkgtest [18:32:17]: build needed for binaries
autopkgtest [18:32:24]: ERROR: "dpkg --unpack /tmp/autopkgtest.4vFMyL/1-autopkgtest-satdep.deb" failed with stderr "dpkg-deb: error: archive '/tmp/autopkgtest.4vFMyL/1-autopkgtest-satdep.deb' uses unknown compression for member 'control.tar.zst', giving up
dpkg: error processing archive /tmp/autopkgtest.4vFMyL/1-autopkgtest-satdep.deb (--unpack):
 dpkg-deb --control subprocess returned error exit status 2
Errors were encountered while processing:
 /tmp/autopkgtest.4vFMyL/1-autopkgtest-satdep.deb

Then, upgrade the package to the version with the proposed fix and re-run the tests

$ autopkgtest ./. -- lxd autopkgtest/debian/sid/amd64

Finally, verify the error no longer occurs.

[Where problems could occur]

We will be enforcing the xz compression format. While this is available in dpkg throughout all supported Ubuntu releases, this could become an issue for targets that do not support such compression format. In this case, the change would need to be reverted or changed to a compression format supported by older versions of dpkg (i.e., gz).

[Other Info]

This have been fixed in jammy on 5.19.

[Original message]

Since impish, dpkg defaults to the zstd compression format for compressing the members in a deb package archive [1,2].

However Debian zstd support efforts are still in progress [3].

Therefore, running autopkgtest from an Ubuntu (>= 21.10) host on a Debian target fails, since autopkgtest uses dpkg-deb to generate the autopkgtest-satdep.deb package.

For instance, running the following commands for a given package

$ autopkgtest-build-lxd images:debian/sid
$ autopkgtest ./. -- lxd autopkgtest/debian/sid/amd64

results in the following error:

autopkgtest [18:32:17]: build needed for binaries
autopkgtest [18:32:24]: ERROR: "dpkg --unpack /tmp/autopkgtest.4vFMyL/1-autopkgtest-satdep.deb" failed with stderr "dpkg-deb: error: archive '/tmp/autopkgtest.4vFMyL/1-autopkgtest-satdep.deb' uses unknown compression for member 'control.tar.zst', giving up
dpkg: error processing archive /tmp/autopkgtest.4vFMyL/1-autopkgtest-satdep.deb (--unpack):
 dpkg-deb --control subprocess returned error exit status 2
Errors were encountered while processing:
 /tmp/autopkgtest.4vFMyL/1-autopkgtest-satdep.deb

[1] https://bugs.launchpad.net/ubuntu/+source/dpkg/+bug/1923845
[2] https://bugs.launchpad.net/ubuntu/+source/dpkg/+bug/1923845/comments/34
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=892664

[Athos Ribeiro (athos-ribeiro)]

A possible solution here would be to force the dpkg-deb operation to use a different compression format until https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=892664 gets sorted out (patch proposal attached).

This would enable developers running Ubuntu to perform autopkgtest tests against Debian before submitting patches there.

A PPA with the proposed patch is available at https://launchpad.net/~athos-ribeiro/+archive/ubuntu/autopkgtest-use-xz/+packages

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "2-5.17ubuntu1.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Benjamin Drung (bdrung)]

I stumbled upon this bug as well, but I would force using gzip instead of xz (for performance reasons).

[Julian Andres Klode (juliank)]

Can you please submit this change upstream?

https://salsa.debian.org/ci-team/autopkgtest

Thank you!

[Julian Andres Klode (juliank)]

Unsubscribed ubuntu sponsors as the patch does not apply anymore.

Forcing xz is fine, there's nothing to compress anyhow, it doesn't really matter.

[Athos Ribeiro (athos-ribeiro)]

I rebased the patch on the newest version and filed a new MP at [1].

The package is available in a PPA at [2].

When I first prepared this patch, I stumbled upon [3], where there was some resistance in hard coding a compression format upstream due to the technical debt it could cause. Specially when we would want to revert the change once [4] is accepted. Still, I am submitting it there as suggested [5].

I ran the dep8 test suite locally, here is the result summary:

On a note for my future self, when reviewing this bug I started wondering (for the second time) why we are only seeing this in Debian an not in older Ubuntu releases. The reason is because zstd support was back-ported all the way down to xenial.

[1] https://code.launchpad.net/~athos-ribeiro/ubuntu/+source/autopkgtest/+git/autopkgtest/+merge/412406
[2] https://launchpad.net/~athos-ribeiro/+archive/ubuntu/autopkgtest-use-xz/+packages
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=883167
[4] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=892664
[5] https://salsa.debian.org/ci-team/autopkgtest/-/merge_requests/133

[Athos Ribeiro (athos-ribeiro)]

FYI: https://salsa.debian.org/ci-team/autopkgtest/-/merge_requests/133 was accepted in Debian

[Benjamin Drung (bdrung)]

Great. So your fix is included in autopkgtest 5.19 which was synced to jammy yesterday: https://launchpad.net/ubuntu/+source/autopkgtest/5.19

[Benjamin Drung (bdrung)]

You might want to get it fixed for impish as well.

[Brian Murray (brian-murray)]

Hello Athos, or anyone else affected,

Accepted autopkgtest into impish-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/autopkgtest/5.16ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-impish to verification-done-impish. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-impish. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Athos Ribeiro (athos-ribeiro)]

I verified this fix by

Launching an impish vm:

$ lxc launch ubuntu-daily:impish autopkgtest-impish --vm
$ lxc exec autopkgtest-impish bash

installing autopkgtest 5.16ubuntu1

# apt update
# apt install -y autopkgtest

Setting up a test environment (I used tmux source package, any package with a dep8 test suite should do it):

# lxd init
# autopkgtest-build-lxd images:debian/sid
# git clone https://salsa.debian.org/rfrancoise/tmux
# cd tmux/

Running the tests in the Debian target:

# autopkgtest ./. -- lxd autopkgtest/debian/sid/amd64

Which confirmed the bug with the following output

autopkgtest [19:35:26]: ERROR: "dpkg --unpack /tmp/autopkgtest.0SmQcL/1-autopkgtest-satdep.deb" failed with stderr "dpkg-deb: error: archive '/tmp/autopkgtest.0SmQcL/1-autopkgtest-satdep.deb' uses unknown compression for member 'control.tar.zst', giving up
dpkg: error processing archive /tmp/autopkgtest.0SmQcL/1-autopkgtest-satdep.deb (--unpack):
 dpkg-deb --control subprocess returned error exit status 2
Errors were encountered while processing:
 /tmp/autopkgtest.0SmQcL/1-autopkgtest-satdep.deb
"

Then, I enabled the -proposed pocket, installed autopkgtest 5.16ubuntu2, and re-ran the tests:

# autopkgtest ./. -- lxd autopkgtest/debian/sid/amd64

Which confirmed the fix by running the test suite:

autopkgtest [19:50:24]: @@@@@@@@@@@@@@@@@@@@ summary
smoketest PASS
utmp PASS

[Launchpad Janitor (janitor)]

This bug was fixed in the package autopkgtest - 5.16ubuntu2

---------------
autopkgtest (5.16ubuntu2) impish; urgency=medium

  * Force xz compression for satdep.deb to support testing on Debian
    environments (LP: #1947810)

 -- Athos Ribeiro <email address hidden> Wed, 15 Dec 2021 12:58:11 -0300

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for autopkgtest has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.