xz compressed control.tar files not supported

[Witold Szczeponik]

[impact]

trusty dpkg fails when trying to extract a dpkg source package compressed using xz. This breaks running autopkgtest from bionic with any and all trusty packages.

[test case]

on a bionic system, get any trusty source dpkg and run autopkgtest with it. See comment 11 for an example.

[regression potential]

any changes to dpkg has the potential to break all package changes on the system, which obviously is a serious regression. However, this specific change has been upstream for a very long time. Any breakage would happen when dpkg is either building or extracting a deb.

[other info]

this affects debootstrap as well, as described in the original bug description:

---

trusty's dpkg (1.17.5ubuntu5.7) cannot handle .deb files that contain control.tar files that are compressed using the xz compressor.

Here's an example of such a case when trying to show the contents of a .deb file from bionic's repositories:

dpkg-deb: error: archive './var/cache/apt/archives/ubuntu-minimal_1.406_amd64.deb' has premature member 'control.tar.xz' before 'control.tar.gz', giving up

This issue makes it currently impossible to create a bionic environment using debootstrap (currently from trusty-proposed).

I have observed the issue on trusty, precisely on

$ lsb_release -rd
Description: Ubuntu 14.04.5 LTS
Release: 14.04

using

$ apt-cache policy dpkg
dpkg:
  Installed: 1.17.5ubuntu5.7
  Candidate: 1.17.5ubuntu5.7
  Version table:
 *** 1.17.5ubuntu5.7 0
        500 mirror://mirrors.ubuntu.com/mirrors.txt/ trusty-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     1.17.5ubuntu5.6 0
        500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
     1.17.5ubuntu5 0
        500 mirror://mirrors.ubuntu.com/mirrors.txt/ trusty/main amd64 Packages

[Witold Szczeponik (wsz)]

Commit c17be3cbfc58e5b54ae1d5ae4714460d7ae2e15c from git://anonscm.debian.org/dpkg/dpkg.git seems to contain a fix.

It can be applied to 1.17.5ubuntu5.7, and I successfully built a package. With said patch applied, xz compressed control.tar files seem to be supported.

[Brian Murray (brian-murray)]

This also affects Precise. Here's a link to the commit:

https://anonscm.debian.org/cgit/dpkg/dpkg.git/commit/?id=c17be3cbfc58e5b54ae1d5ae4714460d7ae2e15c

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Anthony Yarbrough (virento)]

I'm experiencing this issue when attempting to install the latest Liquorix kernel that I've been using for years.

$ lsb_release -rd
Description: Ubuntu 14.04.5 LTS
Release: 14.04

$ apt-cache policy dpkg
dpkg:
  Installed: 1.17.5ubuntu5.7
  Candidate: 1.17.5ubuntu5.7
  Version table:
 *** 1.17.5ubuntu5.7 0
        500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     1.17.5ubuntu5.6 0
        500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
     1.17.5ubuntu5 0

$ sudo apt full-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following NEW packages will be installed:
  linux-headers-4.13.0-13.1-liquorix-amd64
  linux-image-4.13.0-13.1-liquorix-amd64
The following packages will be upgraded:
  linux-headers-liquorix-amd64 linux-image-liquorix-amd64
2 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/57.4 MB of archives.
After this operation, 280 MB of additional disk space will be used.
Do you want to continue? [Y/n]
Supported
Create a snapshot of '/tmp/apt-btrfs-snapshot-mp-21ujpnev/@' in '/tmp/apt-btrfs-snapshot-mp-21ujpnev/@apt-snapshot-2017-11-19_17:41:21'
Preconfiguring packages ...
dpkg-deb: error: archive '/var/cache/apt/archives/linux-image-4.13.0-13.1-liquorix-amd64_4.13-7_amd64.deb' has premature member 'control.tar.xz' before 'control.tar.gz', giving up
dpkg: error processing archive /var/cache/apt/archives/linux-image-4.13.0-13.1-liquorix-amd64_4.13-7_amd64.deb (--unpack):
 subprocess dpkg-deb --control returned error exit status 2
dpkg-deb: error: archive '/var/cache/apt/archives/linux-headers-4.13.0-13.1-liquorix-amd64_4.13-7_amd64.deb' has premature member 'control.tar.xz' before 'control.tar.gz', giving up
dpkg: error processing archive /var/cache/apt/archives/linux-headers-4.13.0-13.1-liquorix-amd64_4.13-7_amd64.deb (--unpack):
 subprocess dpkg-deb --control returned error exit status 2
dpkg-deb: error: archive '/var/cache/apt/archives/linux-headers-liquorix-amd64_4.13-7_amd64.deb' has premature member 'control.tar.xz' before 'control.tar.gz', giving up
dpkg: error processing archive /var/cache/apt/archives/linux-headers-liquorix-amd64_4.13-7_amd64.deb (--unpack):
 subprocess dpkg-deb --control returned error exit status 2
dpkg-deb: error: archive '/var/cache/apt/archives/linux-image-liquorix-amd64_4.13-7_amd64.deb' has premature member 'control.tar.xz' before 'control.tar.gz', giving up
dpkg: error processing archive /var/cache/apt/archives/linux-image-liquorix-amd64_4.13-7_amd64.deb (--unpack):
 subprocess dpkg-deb --control returned error exit status 2
No apport report written because MaxReports is reached already
                                                              Errors were encountered while processing:
 /var/cache/apt/archives/linux-image-4.13.0-13.1-liquorix-amd64_4.13-7_amd64.deb
 /var/cache/apt/archives/linux-headers-4.13.0-13.1-liquorix-amd64_4.13-7_amd64.deb
 /var/cache/apt/archives/linux-headers-liquorix-amd64_4.13-7_amd64.deb
 /var/cache/apt/archives/linux-image-liquorix-amd64_4.13-7_amd64.deb
[ Rootkit Hu...

[Anthony Yarbrough (virento)]

I can confirm that the method described in comment #1 by 'wsz' works like a charm for my Trusty installation and is much appreciated as I can now install the latest Liquorix kernel. Woohoo!

[Robie Basak (racb)]

13:30 <rbasak> infinity: missing SRU paperwork for bug 1730627, but also, do we really need to SRU this? Is being able to debootstrap Bionic from Trusty important?
13:30 <ubottu> bug 1730627 in dpkg (Ubuntu Trusty) "xz compressed control.tar files not supported" [Medium,Confirmed] https://launchpad.net/bugs/1730627
13:31 <rbasak> We don't support an upgrade from Trusty to Bionic directly, so why do we need to support a debootstrap?

[Witold Szczeponik (wsz)]

I am running an environment (currently on Trusty) which supports building packages for the current Ubuntu and Debian releases for a non-trivial user group. For various reasons, I cannot update to Xenial in a timely manner, and since Bionic is looming large, I want (and need) to prepare for that release in a timely manner.

Hence my requirement for a "debootstrap" to Bionic on Trusty. Since a "debootstrap" fails for problems with "dpkg", I created this bug report.

[Witold Szczeponik (wsz)]

Adding to my comment #8: "debootstrap" is not only used for release upgrades, if I am not mistaken. Because if this was the main/only use case, the support for various vanilla Debian releases should not be included in "debootstrap" either.

[Ross Gammon (rosco2)]

Just adding a comment that this is about more than just bootstrapping.

Users on the Trusty LTS cannot install deb packages from upstream websites that were created with xz compression (ie. built with a newer distribution). This is how I came to this bug today.

[Dan Streetman (ddstreet)]

This bug breaks autopkgtest running on bionic for (apparently) all trusty packages.

For example:

$ lsb_release -r
Release: 18.04
$ pull-lp-source bash trusty
...
$ autopkgtest -U bash_4.3-7ubuntu1.7.dsc -- lxd ubuntu-daily:trusty
...
autopkgtest [13:31:50]: ERROR: "dpkg --unpack /tmp/autopkgtest.nywTmz/autopkgtest-satdep.deb" failed with stderr "dpkg-deb: error: archive '/tmp/autopkgtest.nywTmz/autopkgtest-satdep.deb' has premature member 'control.tar.xz' before 'control.tar.gz', giving up

it's like that for all the trusty packages I tested.

[Dan Streetman (ddstreet)]

@infinity I'll get this patch SRUed into trusty.

[Dan Streetman (ddstreet)]

Built a test package with the upstream patch here:
https://launchpad.net/~ddstreet/+archive/ubuntu/lp1730627

Then tested autopkgtest on bionic system, with the patched dpkg. Fails as expected with the current trusty dpkg:

$ autopkgtest -U dpkg_1.17.5ubuntu5.8.dsc -- lxd ubuntu-daily:trusty
...
autopkgtest [15:20:15]: ERROR: "dpkg --unpack /tmp/autopkgtest.wuLHN3/autopkgtest-satdep.deb" failed with stderr "dpkg-deb: error: archive '/tmp/autopkgtest.wuLHN3/autopkgtest-satdep.deb' has premature member 'control.tar.xz' before 'control.tar.gz', giving up

Build/autopkgtest successful with the patched trusty dpkg from PPA:

$ autopkgtest --setup-commands='sudo apt-add-repository ppa:ddstreet/lp1730627' -U dpkg_1.17.5ubuntu5.8.dsc -- lxd ubuntu-daily:trusty
...
dpkg-deb: building package `dpkg-dev' in `../dpkg-dev_1.17.5ubuntu5.8_all.deb'.
 dpkg-genchanges -b >../dpkg_1.17.5ubuntu5.8_amd64.changes
dpkg-genchanges: binary-only upload - not including any source code
 dpkg-source --after-build src
dpkg-source: info: using options from src/debian/source/options: --compression=xz
dpkg-buildpackage: binary only upload (no source included)
dpkg-source: info: using options from src/debian/source/options: --compression=xz
* SKIP no tests in this package
autopkgtest [15:29:22]: @@@@@@@@@@@@@@@@@@@@ summary
* SKIP no tests in this package

[Łukasz Zemczak (sil2100)]

Hello Witold, or anyone else affected,

Accepted dpkg into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dpkg/1.17.5ubuntu5.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-trusty to verification-done-trusty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-trusty. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Dan Streetman (ddstreet)]

verification that it fixes the problem for me, re: autopkgtest. As that was the same cause/problem as initial reporter's, I expect it should be fixed for them as well.

ddstreet@thorin:~/bugs/lp1730627/verification$ pull-lp-source lsb-base trusty
pull-lp-source: Using source package 'lsb' for binary package 'lsb-base'
pull-lp-source: Found lsb 4.1+Debian11ubuntu6.2 in trusty
pull-lp-source: Good signature by Steve Langasek <email address hidden> (0x568D30F321B2133D)
pull-lp-source: Downloading lsb_4.1+Debian11ubuntu6.2.tar.xz from archive.ubuntu.com (0.063 MiB)
dpkg-source: info: extracting lsb in lsb-4.1+Debian11ubuntu6.2
dpkg-source: info: unpacking lsb_4.1+Debian11ubuntu6.2.tar.xz
ddstreet@thorin:~/bugs/lp1730627/verification$ autopkgtest -U lsb_4.1+Debian11ubuntu6.2.dsc -- lxd ubuntu-daily:trusty
autopkgtest [14:16:23]: version 5.1
autopkgtest [14:16:23]: host thorin.lan; command line: /usr/bin/autopkgtest -U lsb_4.1+Debian11ubuntu6.2.dsc -- lxd ubuntu-daily:trusty
...
(normal setup output)
...
autopkgtest [14:17:36]: testbed dpkg architecture: amd64
autopkgtest [14:17:38]: testbed running kernel: Linux 4.15.0-10-generic #11-Ubuntu SMP Tue Feb 13 18:23:35 UTC 2018
autopkgtest [14:17:38]: @@@@@@@@@@@@@@@@@@@@ source lsb_4.1+Debian11ubuntu6.2.dsc
autopkgtest [14:17:42]: ERROR: "dpkg --unpack /tmp/autopkgtest.Mrh8tN/autopkgtest-satdep.deb" failed with stderr "dpkg-deb: error: archive '/tmp/autopkgtest.Mrh8tN/autopkgtest-satdep.deb' has premature member 'control.tar.xz' before 'control.tar.gz', giving up
dpkg: error processing archive /tmp/autopkgtest.Mrh8tN/autopkgtest-satdep.deb (--unpack):
 subprocess dpkg-deb --control returned error exit status 2
Errors were encountered while processing:
 /tmp/autopkgtest.Mrh8tN/autopkgtest-satdep.deb

using -proposed package:

ddstreet@thorin:~/bugs/lp1730627/verification$ autopkgtest --apt-pocket proposed=dpkg -U lsb_4.1+Debian11ubuntu6.2.dsc -- lxd ubuntu-daily:trusty
autopkgtest [14:19:13]: version 5.1
autopkgtest [14:19:13]: host thorin.lan; command line: /usr/bin/autopkgtest --apt-pocket proposed=dpkg -U lsb_4.1+Debian11ubuntu6.2.dsc -- lxd ubuntu-daily:trusty
...
(normal setup output)
...
Preparing to unpack .../dpkg-dev_1.17.5ubuntu5.8_all.deb ...
Unpacking dpkg-dev (1.17.5ubuntu5.8) ...
...
dpkg-buildpackage: binary only upload (no source included)
dpkg-source: info: using options from src/debian/source/options: --compression=xz
* SKIP no tests in this package
autopkgtest [14:24:14]: @@@@@@@@@@@@@@@@@@@@ summary
* SKIP no tests in this package

autopkgtest is able to continue with -proposed dpkg.

[Witold Szczeponik (wsz)]

I can confirm that the proposed package fixes the problem I reported (with sample ubuntu-minimal_1.412_amd64.deb being from bionic's repositories):

$ dpkg-deb -W ./ubuntu-minimal_1.412_amd64.deb
ubuntu-minimal 1.412

$ lsb_release -rd
Description: Ubuntu 14.04.5 LTS
Release: 14.04

$ apt-cache policy dpkg
dpkg:
  Installed: 1.17.5ubuntu5.8
  Candidate: 1.17.5ubuntu5.8
  Version table:
 *** 1.17.5ubuntu5.8 0
        100 /var/lib/dpkg/status
     1.17.5ubuntu5.7 0
        500 http://archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
     1.17.5ubuntu5.6 0
        500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
     1.17.5ubuntu5 0
        500 http://archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages

[Witold Szczeponik (wsz)]

I can also confirm that building a "Bionic" debootstrap environment on "Trusty" works too.

[cleary (bernard-gray)]

@sil2100 Tested, confirmed it's fixed this bug for me - thankyou :)

[Launchpad Janitor (janitor)]

This bug was fixed in the package dpkg - 1.17.5ubuntu5.8

---------------
dpkg (1.17.5ubuntu5.8) trusty; urgency=medium

  * Add support for .deb archives with a control member not compressed
    (control.tar) or compressed with xz (control.tar.xz) LP: #1730627.

 -- Adam Conrad <email address hidden> Mon, 04 Dec 2017 12:15:45 -0700

[Adam Conrad (adconrad)]

The verification of the Stable Release Update for dpkg has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Phil Evans (quantumphil)]

I'm seeing this bug on Ubuntu bionic 18.04 with dpkg v.1.19.0.5ubuntu2.

$ lsb_release -rd
Description: Ubuntu 18.04 LTS
Release: 18.04

$ sudo apt-cache policy dpkg
dpkg:
  Installed: 1.19.0.5ubuntu2
  Candidate: 1.19.0.5ubuntu2
  Version table:
 *** 1.19.0.5ubuntu2 500
        500 http://<internal repo pointing to us.rsync.archive.ubuntu.com/ubuntu/>/ubuntu bionic/main amd64 Packages
        100 /var/lib/dpkg/status

$ sudo dpkg -i <file>.deb
dpkg-deb: error: archive '<file>.deb' has premature member 'data.tar.gz' before 'control.tar', giving up
dpkg: error processing archive <file>.deb (--install):
 dpkg-deb --control subprocess returned error exit status 2
Errors were encountered while processing:
 <file>.deb

[Dan Streetman (ddstreet)]

> I'm seeing this bug on Ubuntu bionic 18.04 with dpkg v.1.19.0.5ubuntu2.

no, you're not. The message may look similar but it is not the same.

> $ sudo dpkg -i <file>.deb

hm, i don't have the '<file>.deb' anywhere on my system ;-P

Obfuscating what deb you're have a problem with isn't helpful as the problem very well may be with that deb, and makes it impossible for anyone to reproduce your problem.

[Adonis Tarcio Moreira (adonist)]

I'm having the same issue with the package check-mk-agent_1.5.0b9-1_all.deb from Nagios (Check_MK).

# lsb_release -rd
Description: Ubuntu 18.04.1 LTS
Release: 18.04

# apt-cache policy dpkg
dpkg:
  Installed: 1.19.0.5ubuntu2.1
  Candidate: 1.19.0.5ubuntu2.1
  Version table:
 *** 1.19.0.5ubuntu2.1 500
        500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     1.19.0.5ubuntu2 500
        500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages

# dpkg -i check-mk-agent_1.5.0b9-1_all.deb
(Reading database ... 239815 files and directories currently installed.)
Preparing to unpack check-mk-agent_1.5.0b9-1_all.deb ...
Unpacking check-mk-agent (1.5.0b9-1) ...
dpkg-deb: error: archive 'check-mk-agent_1.5.0b9-1_all.deb' contains two control members, giving up
dpkg: error processing archive check-mk-agent_1.5.0b9-1_all.deb (--install):
 dpkg-deb --fsys-tarfile subprocess returned error exit status 2
Reloading xinetd...
Errors were encountered while processing:
 check-mk-agent_1.5.0b9-1_all.deb

[Dan Streetman (ddstreet)]

> I'm having the same issue with the package check-mk-agent_1.5.0b9-1_all.deb from
> Nagios (Check_MK).

The problem is with your deb, not dpkg. The check-mk-agent deb from Ubuntu works fine:

$ apt-cache policy check-mk-agent
check-mk-agent:
  Installed: 1.2.8p16-1ubuntu0.1
  Candidate: 1.2.8p16-1ubuntu0.1
  Version table:
 *** 1.2.8p16-1ubuntu0.1 500
        500 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages
        100 /var/lib/dpkg/status

$ sudo dpkg -i check-mk-agent_1.2.8p16-1ubuntu0.1_amd64.deb
Selecting previously unselected package check-mk-agent.
(Reading database ... 40378 files and directories currently installed.)
Preparing to unpack check-mk-agent_1.2.8p16-1ubuntu0.1_amd64.deb ...
Unpacking check-mk-agent (1.2.8p16-1ubuntu0.1) ...
Setting up check-mk-agent (1.2.8p16-1ubuntu0.1) ...

This is not the bug you are looking for.