Broken juju LXC deployments

[Brad Marshall]

I've just tried using juju to deploy to a container with trusty-proposed repo enabled, and I get an error message about 'failed to retrieve the template to clone'. The underlying error appears to be:

  tar --numeric-owner -xpJf /var/cache/lxc/cloud-trusty/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz;
  xz: (stdin): File format not recognized; tar: Child returned status 1; tar:
  Error is not recoverable: exiting now;

This seems to be fairly obvious, trying to use xz on a tar.gz file is never going to work.

The change appears to be from https://github.com/lxc/lxc/commit/27c278a76931bfc4660caa85d1942ca91c86e0bf, it assumes everything passed into it will be a .tar.xz file.

This appears to be a conflict between the template expecting a .tar.xz file, and juju providing it a .tar.gz file. You can see what juju is providing from:

  $ ubuntu-cloudimg-query trusty released amd64 --format %{url}
  https://cloud-images.ubuntu.com/server/releases/trusty/release-20151105/ubuntu-14.04-server-cloudimg-amd64.tar.gz

From the juju deployed host:
$ apt-cache policy lxc-templates
lxc-templates:
  Installed: 1.0.8-0ubuntu0.1
  Candidate: 1.0.8-0ubuntu0.1
  Version table:
 *** 1.0.8-0ubuntu0.1 0
        500 http://archive.ubuntu.com/ubuntu/ trusty-proposed/main amd64 Packages
        100 /var/lib/dpkg/status

From the host running juju:
$ apt-cache policy juju-core
juju-core:
  Installed: 1.22.8-0ubuntu1~14.04.1
  Candidate: 1.25.0-0ubuntu1~14.04.1~juju1
  Version table:
     1.25.0-0ubuntu1~14.04.1~juju1 0
        500 http://ppa.launchpad.net/juju/proposed/ubuntu/ trusty/main amd64 Packages
 *** 1.22.8-0ubuntu1~14.04.1 0
        400 http://archive.ubuntu.com/ubuntu/ trusty-proposed/universe amd64 Packages
        100 /var/lib/dpkg/status

All machine involved are running trusty:

$ lsb_release -rd
Description: Ubuntu 14.04.3 LTS
Release: 14.04

Please let me know if you need any more information.

[Stéphane Graber (stgraber)]

Oh yeah, I can see how this is going to be a problem.

I'll look at changing the code path to use auto-detected compression rather than specific one.

The reason for this change in the first place is that the .tar.gz images will be deprecated in favor for .tar.xz, so I'd recommend JuJu also switch to .tar.xz at some point soon (I don't believe the cloud team has set a timeframe for phasing out tar.gz yet, but it's planned).

[Ian Booth (wallyworld)]

Stephane,

Juju doesn't decide what images to use directly. It uses ubuntu-cloudimg-query for that.
So this utility will need to be changed. It is in the cloud-image-utils package.

[Stéphane Graber (stgraber)]

Well, the string returned by ubuntu-cloudimg-query already requires mangling to be usable with the lxc-ubuntu-cloud template (as we do inside the ubuntu-cloud template).

As there are no full cloud images compressed using .tar.xz, ubuntu-cloudimg-query will continue to return .tar.gz and it's up to whatever piece of software currently mangles its return value to the .root.tar.gz equivalent to instead mangle it to .root.tar.xz (as I did in the ubuntu-cloud template).

Anyway, as I said, I'll be changing the tar commands so they don't care about the compression algorithm, that should make transitioning much easier.

[Ian Booth (wallyworld)]

Stephane, you are right. We can change Juju to use mangle the output of the cloud-image-query to add "root.tar.xz". Is this guaranteed to always work for all series / arch? Are there the correct root.tar.xz files on cloud-image.ubuntu.com ?

[Stéphane Graber (stgraber)]

root.tar.xz images are available for all series and all architectures and are indeed on cloud-images.ubuntu.com

I've not checked them all myself but after the CPC team told me they added them to their generated images, I did confirm (through LXD which uses them) that at least precise, trusty, vivid, wily and xenial are all available and working on x86 and armhf.

[Stéphane Graber (stgraber)]

Note that the only different between root.tar.gz and root.tar.xz is the last compression step, the uncompressed tarball is the same.

[Brad Marshall (brad-marshall)]

FWIW and a totally expected result, I just downgraded the LXC packages on these hosts and redeployed, and things came up ok.

$ dpkg-query -W lxc
lxc 1.0.7-0ubuntu0.10

I don't think this changes anything, but just putting it here for completeness.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Stéphane Graber (stgraber)]

I've sent a few upstream now and will push it to all releases as soon as it's acked.

[Stéphane Graber (stgraber)]

That was "a few changes" :)

The patch was acked and was pushed to all upstream stable branches. I'm waiting for CI to confirm it all looks good and I'll be uploading packages in the morning.

[Stéphane Graber (stgraber)]

Uploaded the fix to xenial, wily, vivid and trusty.

[Chris J Arges (arges)]

Hello Brad, or anyone else affected,

Accepted lxc into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lxc/1.1.5-0ubuntu0.15.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris J Arges (arges)]

Hello Brad, or anyone else affected,

Accepted lxc into wily-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lxc/1.1.5-0ubuntu0.15.10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris J Arges (arges)]

Hello Brad, or anyone else affected,

Accepted lxc into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lxc/1.0.8-0ubuntu0.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Launchpad Janitor (janitor)]

This bug was fixed in the package lxc - 1.1.5-0ubuntu2

---------------
lxc (1.1.5-0ubuntu2) xenial; urgency=medium

  * Cherry-pick from upstream:
    - Fix ubuntu-cloud template to detect compression algorithm instead
      of hardcoding xz. Also update list of supported releases and use trusty
      as the fallback release. (LP: #1515463)
  * Update lxc-tests description to make it clear that this package is
    meant to be used by developers and by automated testing.

 -- Stéphane Graber <email address hidden> Fri, 13 Nov 2015 12:05:36 -0500

[Chris J Arges (arges)]

Hello Brad, or anyone else affected,

Accepted lxc into wily-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lxc/1.1.5-0ubuntu0.15.10.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris J Arges (arges)]

Hello Brad, or anyone else affected,

Accepted lxc into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lxc/1.1.5-0ubuntu0.15.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris J Arges (arges)]

Hello Brad, or anyone else affected,

Accepted lxc into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lxc/1.0.8-0ubuntu0.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Stéphane Graber (stgraber)]

Brad: any chance you can confirm this works as expected?

[Brad Marshall (brad-marshall)]

This does indeed appear to work correctly, I've deployed a container using juju:

ubuntu@apollo:~$ dpkg-query -W lxc
lxc 1.0.8-0ubuntu0.3

ubuntu@apollo:~$ sudo lxc-ls --fancy
NAME STATE IPV4 IPV6 AUTOSTART
------------------------------------------------------------------
juju-machine-0-lxc-0 RUNNING x.y.z.171 - YES
juju-trusty-lxc-template STOPPED - - NO

ubuntu@apollo:~$ sudo lxc-attach -n juju-machine-0-lxc-0
root@juju-machine-0-lxc-0:~#

Thanks!

[Stéphane Graber (stgraber)]

Great. Current plan is to release the SRUs on Monday, I figured releasing them over the thanksgiving weekend wouldn't be the best idea.

[Launchpad Janitor (janitor)]

This bug was fixed in the package lxc - 1.0.8-0ubuntu0.3

---------------
lxc (1.0.8-0ubuntu0.3) trusty; urgency=medium

  * Cherry-pick from upstream:
    - Fix preserve_ns to work on < 3.8 kernels. (LP: #1516971)

lxc (1.0.8-0ubuntu0.2) trusty; urgency=medium

  * Cherry-pick from upstream:
    - Fix ubuntu-cloud template to detect compression algorithm instead
      of hardcoding xz. Also update list of supported releases and use trusty
      as the fallback release. (LP: #1515463)
  * Update lxc-tests description to make it clear that this package is
    meant to be used by developers and by automated testing.

lxc (1.0.8-0ubuntu0.1) trusty; urgency=medium

  * New upstream bugfix release. (MRE tracking bug: LP: #1514623)
    (LP: #1429140)
    - Changelog at: https://linuxcontainers.org/lxc/news/
  * Drop proxy detection from the autopkgtest exercise script.
  * Add patch:
    - 0001-Trusty-Swap-out-the-CVE-2015-1335-fix-with-the-trust.patch
      This is a patch by Serge Hallyn to cope with the trusty 3.13 kernel.
      It updates the upstream CVE fix to the version which trusty ended
      up with after the few round of fixes.

 -- Stéphane Graber <email address hidden> Wed, 18 Nov 2015 13:42:07 -0500

[Stéphane Graber (stgraber)]

The verification of the Stable Release Update for lxc has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package lxc - 1.1.5-0ubuntu0.15.04.3

---------------
lxc (1.1.5-0ubuntu0.15.04.3) vivid-proposed; urgency=medium

  * Cherry-pick from upstream:
    - Fix preserve_ns to work on < 3.8 kernels. (LP: #1516971)
    - Fix process title rewrite to not mangle the environment. (LP: #1517107)

lxc (1.1.5-0ubuntu0.15.04.2) vivid-proposed; urgency=medium

  * Cherry-pick from upstream:
    - Fix ubuntu-cloud template to detect compression algorithm instead
      of hardcoding xz. Also update list of supported releases and use trusty
      as the fallback release. (LP: #1515463)
  * Update lxc-tests description to make it clear that this package is
    meant to be used by developers and by automated testing.

lxc (1.1.5-0ubuntu0.15.04.1) vivid-proposed; urgency=medium

  * New upstream bugfix release (MRE) (1.1.5)
    (LP: #1497420, LP: #1436723, LP: #1441068, LP: #1504496,
     LP: #1466458, LP: #1510619)
  * Drop proxy detection from the autopkgtest exercise script.

 -- Stéphane Graber <email address hidden> Wed, 18 Nov 2015 13:41:23 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package lxc - 1.1.5-0ubuntu0.15.10.3

---------------
lxc (1.1.5-0ubuntu0.15.10.3) wily-proposed; urgency=medium

  * Cherry-pick from upstream:
    - Fix preserve_ns to work on < 3.8 kernels. (LP: #1516971)
    - Fix process title rewrite to not mangle the environment. (LP: #1517107)

lxc (1.1.5-0ubuntu0.15.10.2) wily-proposed; urgency=medium

  * Cherry-pick from upstream:
    - Fix ubuntu-cloud template to detect compression algorithm instead
      of hardcoding xz. Also update list of supported releases and use trusty
      as the fallback release. (LP: #1515463)
  * Update lxc-tests description to make it clear that this package is
    meant to be used by developers and by automated testing.

lxc (1.1.5-0ubuntu0.15.10.1) wily-proposed; urgency=medium

  * New upstream bugfix release (MRE) (1.1.5)
    (LP: #1497420, LP: #1441068, LP: #1466458, LP: #1510619)
  * Drop proxy detection from the autopkgtest exercise script.

 -- Stéphane Graber <email address hidden> Wed, 18 Nov 2015 13:40:28 -0500