xz compressed control.tar files not supported

Bug #1730627 reported by Witold Szczeponik on 2017-11-07
66
This bug affects 12 people
Affects Status Importance Assigned to Milestone
dpkg (Ubuntu)
Undecided
Unassigned
Precise
Medium
Unassigned
Trusty
Medium
Adam Conrad

Bug Description

[impact]

trusty dpkg fails when trying to extract a dpkg source package compressed using xz. This breaks running autopkgtest from bionic with any and all trusty packages.

[test case]

on a bionic system, get any trusty source dpkg and run autopkgtest with it. See comment 11 for an example.

[regression potential]

any changes to dpkg has the potential to break all package changes on the system, which obviously is a serious regression. However, this specific change has been upstream for a very long time. Any breakage would happen when dpkg is either building or extracting a deb.

[other info]

this affects debootstrap as well, as described in the original bug description:

---

trusty's dpkg (1.17.5ubuntu5.7) cannot handle .deb files that contain control.tar files that are compressed using the xz compressor.

Here's an example of such a case when trying to show the contents of a .deb file from bionic's repositories:

dpkg-deb: error: archive './var/cache/apt/archives/ubuntu-minimal_1.406_amd64.deb' has premature member 'control.tar.xz' before 'control.tar.gz', giving up

This issue makes it currently impossible to create a bionic environment using debootstrap (currently from trusty-proposed).

I have observed the issue on trusty, precisely on

$ lsb_release -rd
Description: Ubuntu 14.04.5 LTS
Release: 14.04

using

$ apt-cache policy dpkg
dpkg:
  Installed: 1.17.5ubuntu5.7
  Candidate: 1.17.5ubuntu5.7
  Version table:
 *** 1.17.5ubuntu5.7 0
        500 mirror://mirrors.ubuntu.com/mirrors.txt/ trusty-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     1.17.5ubuntu5.6 0
        500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
     1.17.5ubuntu5 0
        500 mirror://mirrors.ubuntu.com/mirrors.txt/ trusty/main amd64 Packages

Witold Szczeponik (wsz) wrote :

Commit c17be3cbfc58e5b54ae1d5ae4714460d7ae2e15c from git://anonscm.debian.org/dpkg/dpkg.git seems to contain a fix.

It can be applied to 1.17.5ubuntu5.7, and I successfully built a package. With said patch applied, xz compressed control.tar files seem to be supported.

Witold Szczeponik (wsz) on 2017-11-07
tags: added: trusty
Brian Murray (brian-murray) wrote :
Changed in dpkg (Ubuntu):
status: New → Fix Released
Changed in dpkg (Ubuntu Precise):
importance: Undecided → Medium
Changed in dpkg (Ubuntu Trusty):
importance: Undecided → Medium
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in dpkg (Ubuntu Precise):
status: New → Confirmed
Changed in dpkg (Ubuntu Trusty):
status: New → Confirmed
Anthony Yarbrough (virento) wrote :
Download full text (3.2 KiB)

I'm experiencing this issue when attempting to install the latest Liquorix kernel that I've been using for years.

$ lsb_release -rd
Description: Ubuntu 14.04.5 LTS
Release: 14.04

$ apt-cache policy dpkg
dpkg:
  Installed: 1.17.5ubuntu5.7
  Candidate: 1.17.5ubuntu5.7
  Version table:
 *** 1.17.5ubuntu5.7 0
        500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     1.17.5ubuntu5.6 0
        500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
     1.17.5ubuntu5 0

$ sudo apt full-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following NEW packages will be installed:
  linux-headers-4.13.0-13.1-liquorix-amd64
  linux-image-4.13.0-13.1-liquorix-amd64
The following packages will be upgraded:
  linux-headers-liquorix-amd64 linux-image-liquorix-amd64
2 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/57.4 MB of archives.
After this operation, 280 MB of additional disk space will be used.
Do you want to continue? [Y/n]
Supported
Create a snapshot of '/tmp/apt-btrfs-snapshot-mp-21ujpnev/@' in '/tmp/apt-btrfs-snapshot-mp-21ujpnev/@apt-snapshot-2017-11-19_17:41:21'
Preconfiguring packages ...
dpkg-deb: error: archive '/var/cache/apt/archives/linux-image-4.13.0-13.1-liquorix-amd64_4.13-7_amd64.deb' has premature member 'control.tar.xz' before 'control.tar.gz', giving up
dpkg: error processing archive /var/cache/apt/archives/linux-image-4.13.0-13.1-liquorix-amd64_4.13-7_amd64.deb (--unpack):
 subprocess dpkg-deb --control returned error exit status 2
dpkg-deb: error: archive '/var/cache/apt/archives/linux-headers-4.13.0-13.1-liquorix-amd64_4.13-7_amd64.deb' has premature member 'control.tar.xz' before 'control.tar.gz', giving up
dpkg: error processing archive /var/cache/apt/archives/linux-headers-4.13.0-13.1-liquorix-amd64_4.13-7_amd64.deb (--unpack):
 subprocess dpkg-deb --control returned error exit status 2
dpkg-deb: error: archive '/var/cache/apt/archives/linux-headers-liquorix-amd64_4.13-7_amd64.deb' has premature member 'control.tar.xz' before 'control.tar.gz', giving up
dpkg: error processing archive /var/cache/apt/archives/linux-headers-liquorix-amd64_4.13-7_amd64.deb (--unpack):
 subprocess dpkg-deb --control returned error exit status 2
dpkg-deb: error: archive '/var/cache/apt/archives/linux-image-liquorix-amd64_4.13-7_amd64.deb' has premature member 'control.tar.xz' before 'control.tar.gz', giving up
dpkg: error processing archive /var/cache/apt/archives/linux-image-liquorix-amd64_4.13-7_amd64.deb (--unpack):
 subprocess dpkg-deb --control returned error exit status 2
No apport report written because MaxReports is reached already
                                                              Errors were encountered while processing:
 /var/cache/apt/archives/linux-image-4.13.0-13.1-liquorix-amd64_4.13-7_amd64.deb
 /var/cache/apt/archives/linux-headers-4.13.0-13.1-liquorix-amd64_4.13-7_amd64.deb
 /var/cache/apt/archives/linux-headers-liquorix-amd64_4.13-7_amd64.deb
 /var/cache/apt/archives/linux-image-liquorix-amd64_4.13-7_amd64.deb
[ Rootkit Hu...

Read more...

Adam Conrad (adconrad) on 2017-12-04
Changed in dpkg (Ubuntu Precise):
status: Confirmed → Won't Fix
Anthony Yarbrough (virento) wrote :

I can confirm that the method described in comment #1 by 'wsz' works like a charm for my Trusty installation and is much appreciated as I can now install the latest Liquorix kernel. Woohoo!

Adam Conrad (adconrad) on 2017-12-06
Changed in dpkg (Ubuntu Trusty):
assignee: nobody → Adam Conrad (adconrad)
Robie Basak (racb) wrote :

13:30 <rbasak> infinity: missing SRU paperwork for bug 1730627, but also, do we really need to SRU this? Is being able to debootstrap Bionic from Trusty important?
13:30 <ubottu> bug 1730627 in dpkg (Ubuntu Trusty) "xz compressed control.tar files not supported" [Medium,Confirmed] https://launchpad.net/bugs/1730627
13:31 <rbasak> We don't support an upgrade from Trusty to Bionic directly, so why do we need to support a debootstrap?

Witold Szczeponik (wsz) wrote :

I am running an environment (currently on Trusty) which supports building packages for the current Ubuntu and Debian releases for a non-trivial user group. For various reasons, I cannot update to Xenial in a timely manner, and since Bionic is looming large, I want (and need) to prepare for that release in a timely manner.

Hence my requirement for a "debootstrap" to Bionic on Trusty. Since a "debootstrap" fails for problems with "dpkg", I created this bug report.

Witold Szczeponik (wsz) wrote :

Adding to my comment #8: "debootstrap" is not only used for release upgrades, if I am not mistaken. Because if this was the main/only use case, the support for various vanilla Debian releases should not be included in "debootstrap" either.

Ross Gammon (rosco2) wrote :

Just adding a comment that this is about more than just bootstrapping.

Users on the Trusty LTS cannot install deb packages from upstream websites that were created with xz compression (ie. built with a newer distribution). This is how I came to this bug today.

Dan Streetman (ddstreet) wrote :

This bug breaks autopkgtest running on bionic for (apparently) all trusty packages.

For example:

$ lsb_release -r
Release: 18.04
$ pull-lp-source bash trusty
...
$ autopkgtest -U bash_4.3-7ubuntu1.7.dsc -- lxd ubuntu-daily:trusty
...
autopkgtest [13:31:50]: ERROR: "dpkg --unpack /tmp/autopkgtest.nywTmz/autopkgtest-satdep.deb" failed with stderr "dpkg-deb: error: archive '/tmp/autopkgtest.nywTmz/autopkgtest-satdep.deb' has premature member 'control.tar.xz' before 'control.tar.gz', giving up

it's like that for all the trusty packages I tested.

Changed in dpkg (Ubuntu Trusty):
assignee: Adam Conrad (adconrad) → Dan Streetman (ddstreet)
Dan Streetman (ddstreet) wrote :

@infinity I'll get this patch SRUed into trusty.

Dan Streetman (ddstreet) wrote :

Built a test package with the upstream patch here:
https://launchpad.net/~ddstreet/+archive/ubuntu/lp1730627

Then tested autopkgtest on bionic system, with the patched dpkg. Fails as expected with the current trusty dpkg:

$ autopkgtest -U dpkg_1.17.5ubuntu5.8.dsc -- lxd ubuntu-daily:trusty
...
autopkgtest [15:20:15]: ERROR: "dpkg --unpack /tmp/autopkgtest.wuLHN3/autopkgtest-satdep.deb" failed with stderr "dpkg-deb: error: archive '/tmp/autopkgtest.wuLHN3/autopkgtest-satdep.deb' has premature member 'control.tar.xz' before 'control.tar.gz', giving up

Build/autopkgtest successful with the patched trusty dpkg from PPA:

$ autopkgtest --setup-commands='sudo apt-add-repository ppa:ddstreet/lp1730627' -U dpkg_1.17.5ubuntu5.8.dsc -- lxd ubuntu-daily:trusty
...
dpkg-deb: building package `dpkg-dev' in `../dpkg-dev_1.17.5ubuntu5.8_all.deb'.
 dpkg-genchanges -b >../dpkg_1.17.5ubuntu5.8_amd64.changes
dpkg-genchanges: binary-only upload - not including any source code
 dpkg-source --after-build src
dpkg-source: info: using options from src/debian/source/options: --compression=xz
dpkg-buildpackage: binary only upload (no source included)
dpkg-source: info: using options from src/debian/source/options: --compression=xz
* SKIP no tests in this package
autopkgtest [15:29:22]: @@@@@@@@@@@@@@@@@@@@ summary
* SKIP no tests in this package

Dan Streetman (ddstreet) on 2018-02-23
description: updated
Adam Conrad (adconrad) on 2018-02-23
Changed in dpkg (Ubuntu Trusty):
assignee: Dan Streetman (ddstreet) → Adam Conrad (adconrad)

Hello Witold, or anyone else affected,

Accepted dpkg into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dpkg/1.17.5ubuntu5.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-trusty to verification-done-trusty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-trusty. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in dpkg (Ubuntu Trusty):
status: Confirmed → Fix Committed
tags: added: verification-needed verification-needed-trusty
Dan Streetman (ddstreet) wrote :

verification that it fixes the problem for me, re: autopkgtest. As that was the same cause/problem as initial reporter's, I expect it should be fixed for them as well.

ddstreet@thorin:~/bugs/lp1730627/verification$ pull-lp-source lsb-base trusty
pull-lp-source: Using source package 'lsb' for binary package 'lsb-base'
pull-lp-source: Found lsb 4.1+Debian11ubuntu6.2 in trusty
pull-lp-source: Good signature by Steve Langasek <email address hidden> (0x568D30F321B2133D)
pull-lp-source: Downloading lsb_4.1+Debian11ubuntu6.2.tar.xz from archive.ubuntu.com (0.063 MiB)
dpkg-source: info: extracting lsb in lsb-4.1+Debian11ubuntu6.2
dpkg-source: info: unpacking lsb_4.1+Debian11ubuntu6.2.tar.xz
ddstreet@thorin:~/bugs/lp1730627/verification$ autopkgtest -U lsb_4.1+Debian11ubuntu6.2.dsc -- lxd ubuntu-daily:trusty
autopkgtest [14:16:23]: version 5.1
autopkgtest [14:16:23]: host thorin.lan; command line: /usr/bin/autopkgtest -U lsb_4.1+Debian11ubuntu6.2.dsc -- lxd ubuntu-daily:trusty
...
(normal setup output)
...
autopkgtest [14:17:36]: testbed dpkg architecture: amd64
autopkgtest [14:17:38]: testbed running kernel: Linux 4.15.0-10-generic #11-Ubuntu SMP Tue Feb 13 18:23:35 UTC 2018
autopkgtest [14:17:38]: @@@@@@@@@@@@@@@@@@@@ source lsb_4.1+Debian11ubuntu6.2.dsc
autopkgtest [14:17:42]: ERROR: "dpkg --unpack /tmp/autopkgtest.Mrh8tN/autopkgtest-satdep.deb" failed with stderr "dpkg-deb: error: archive '/tmp/autopkgtest.Mrh8tN/autopkgtest-satdep.deb' has premature member 'control.tar.xz' before 'control.tar.gz', giving up
dpkg: error processing archive /tmp/autopkgtest.Mrh8tN/autopkgtest-satdep.deb (--unpack):
 subprocess dpkg-deb --control returned error exit status 2
Errors were encountered while processing:
 /tmp/autopkgtest.Mrh8tN/autopkgtest-satdep.deb

using -proposed package:

ddstreet@thorin:~/bugs/lp1730627/verification$ autopkgtest --apt-pocket proposed=dpkg -U lsb_4.1+Debian11ubuntu6.2.dsc -- lxd ubuntu-daily:trusty
autopkgtest [14:19:13]: version 5.1
autopkgtest [14:19:13]: host thorin.lan; command line: /usr/bin/autopkgtest --apt-pocket proposed=dpkg -U lsb_4.1+Debian11ubuntu6.2.dsc -- lxd ubuntu-daily:trusty
...
(normal setup output)
...
Preparing to unpack .../dpkg-dev_1.17.5ubuntu5.8_all.deb ...
Unpacking dpkg-dev (1.17.5ubuntu5.8) ...
...
dpkg-buildpackage: binary only upload (no source included)
dpkg-source: info: using options from src/debian/source/options: --compression=xz
* SKIP no tests in this package
autopkgtest [14:24:14]: @@@@@@@@@@@@@@@@@@@@ summary
* SKIP no tests in this package

autopkgtest is able to continue with -proposed dpkg.

tags: added: verification-done verification-done-trusty
removed: verification-needed verification-needed-trusty
Witold Szczeponik (wsz) wrote :

I can confirm that the proposed package fixes the problem I reported (with sample ubuntu-minimal_1.412_amd64.deb being from bionic's repositories):

$ dpkg-deb -W ./ubuntu-minimal_1.412_amd64.deb
ubuntu-minimal 1.412

$ lsb_release -rd
Description: Ubuntu 14.04.5 LTS
Release: 14.04

$ apt-cache policy dpkg
dpkg:
  Installed: 1.17.5ubuntu5.8
  Candidate: 1.17.5ubuntu5.8
  Version table:
 *** 1.17.5ubuntu5.8 0
        100 /var/lib/dpkg/status
     1.17.5ubuntu5.7 0
        500 http://archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
     1.17.5ubuntu5.6 0
        500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
     1.17.5ubuntu5 0
        500 http://archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages

Witold Szczeponik (wsz) wrote :

I can also confirm that building a "Bionic" debootstrap environment on "Trusty" works too.

cleary (bernard-gray) wrote :

@sil2100 Tested, confirmed it's fixed this bug for me - thankyou :)

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package dpkg - 1.17.5ubuntu5.8

---------------
dpkg (1.17.5ubuntu5.8) trusty; urgency=medium

  * Add support for .deb archives with a control member not compressed
    (control.tar) or compressed with xz (control.tar.xz) LP: #1730627.

 -- Adam Conrad <email address hidden> Mon, 04 Dec 2017 12:15:45 -0700

Changed in dpkg (Ubuntu Trusty):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for dpkg has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Phil Evans (quantumphil) wrote :

I'm seeing this bug on Ubuntu bionic 18.04 with dpkg v.1.19.0.5ubuntu2.

$ lsb_release -rd
Description: Ubuntu 18.04 LTS
Release: 18.04

$ sudo apt-cache policy dpkg
dpkg:
  Installed: 1.19.0.5ubuntu2
  Candidate: 1.19.0.5ubuntu2
  Version table:
 *** 1.19.0.5ubuntu2 500
        500 http://<internal repo pointing to us.rsync.archive.ubuntu.com/ubuntu/>/ubuntu bionic/main amd64 Packages
        100 /var/lib/dpkg/status

$ sudo dpkg -i <file>.deb
dpkg-deb: error: archive '<file>.deb' has premature member 'data.tar.gz' before 'control.tar', giving up
dpkg: error processing archive <file>.deb (--install):
 dpkg-deb --control subprocess returned error exit status 2
Errors were encountered while processing:
 <file>.deb

Dan Streetman (ddstreet) wrote :

> I'm seeing this bug on Ubuntu bionic 18.04 with dpkg v.1.19.0.5ubuntu2.

no, you're not. The message may look similar but it is not the same.

> $ sudo dpkg -i <file>.deb

hm, i don't have the '<file>.deb' anywhere on my system ;-P

Obfuscating what deb you're have a problem with isn't helpful as the problem very well may be with that deb, and makes it impossible for anyone to reproduce your problem.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers