juju-core

failure to retrieve the template to clone: lxc container with 1.22 beta2

Bug #1417594 reported by Larry Michel on 2015-02-03
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
​ juju-core
Fix Released
High
Ian Booth
juju-core 1.23-beta1
​ 1.22
Fix Released
High
Ian Booth
juju-core 1.22-beta3
Also affects project (?) Also affects distribution/package Nominate for series

Bug Description

We're seeing this issue on all the nodes when using 1.22 beta2:

From console:
==================================================================
2015-02-03 12:49:29,392 [DEBUG] deployer.import: Waiting for units before adding relations
2015-02-03 12:49:45,539 [DEBUG] deployer.env: Delta machine: 5 change:pending
2015-02-03 12:50:05,541 [DEBUG] deployer.env: Delta machine: 4 change:pending
2015-02-03 12:50:05,544 [DEBUG] deployer.env: Delta machine: 5 change:pending
2015-02-03 12:53:25,566 [DEBUG] deployer.env: Delta machine: 2 change:pending
2015-02-03 12:53:25,570 [DEBUG] deployer.env: Delta machine: 2 change:started
2015-02-03 12:53:25,570 [DEBUG] deployer.env: Delta machine: 3 change:pending
2015-02-03 12:53:25,570 [DEBUG] deployer.env: Delta unit: nova-cloud-controller/0 change:allocating
2015-02-03 12:53:30,569 [DEBUG] deployer.env: Delta machine: 3 change:pending
2015-02-03 12:53:30,573 [DEBUG] deployer.env: Delta machine: 3 change:started
2015-02-03 12:53:30,576 [DEBUG] deployer.env: Delta unit: nova-cloud-controller/0 change:installing
2015-02-03 12:53:30,581 [DEBUG] deployer.env: Delta unit: swift-storage/0 change:installing
2015-02-03 12:53:30,581 [DEBUG] deployer.env: Delta unit: nova-vmware/0 change:installing
2015-02-03 12:53:35,581 [DEBUG] deployer.env: Delta machine: 2/lxc/0 change:error
2015-02-03 12:53:40,582 [DEBUG] deployer.env: Delta machine: 3/lxc/0 change:error
2015-02-03 12:53:40,585 [DEBUG] deployer.env: Delta machine: 3/lxc/1 change:error
2015-02-03 12:54:40,597 [DEBUG] deployer.env: Delta unit: nova-cloud-controller/0 change:installing
2015-02-03 12:58:05,621 [DEBUG] deployer.env: Delta machine: 1 change:pending
2015-02-03 12:58:05,624 [DEBUG] deployer.env: Delta machine: 1 change:started
2015-02-03 12:58:05,624 [DEBUG] deployer.env: Delta unit: cinder/0 change:allocating
2015-02-03 12:58:10,623 [DEBUG] deployer.env: Delta unit: cinder/0 change:installing
2015-02-03 12:58:10,625 [DEBUG] deployer.env: Delta unit: cinder/0 change:installing
2015-02-03 12:58:40,628 [DEBUG] deployer.env: Delta machine: 1/lxc/0 change:error
2015-02-03 12:58:45,629 [DEBUG] deployer.env: Delta machine: 1/lxc/1 change:error
2015-02-03 12:59:50,637 [DEBUG] deployer.env: Delta machine: 5 change:pending
2015-02-03 12:59:50,641 [DEBUG] deployer.env: Delta machine: 5 change:started
2015-02-03 12:59:50,641 [DEBUG] deployer.env: Delta unit: swift-storage/2 change:installing
2015-02-03 13:00:10,642 [DEBUG] deployer.env: Delta unit: nova-vmware/0 change:error
2015-02-03 13:00:10,643 [ERROR] deployer.env: The following units had errors:
   unit: nova-vmware/0: machine: 3 agent-state: error details: hook failed: "install"
2015-02-03 13:00:10,646 [ERROR] oil_ci.deploy.oil_deployer: Deployment failed:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/oil_ci/deploy/oil_deployer.py", line 140, in deploy
    debug=True, verbose=True)
  File "/usr/lib/python2.7/dist-packages/oil_ci/juju/juju_deployer.py", line 85, in run_deployer
    importer.Importer(env, deploy, options=opts).run()
  File "/usr/lib/python2.7/dist-packages/deployer/action/importer.py", line 206, in run
    self.wait_for_units(ignore_errors=ignore_errors)
  File "/usr/lib/python2.7/dist-packages/deployer/action/importer.py", line 181, in wait_for_units
    services=self.deployment.get_service_names(), on_errors=on_errors)
  File "/usr/lib/python2.7/dist-packages/deployer/env/go.py", line 227, in wait_for_units
    services=services, on_errors=on_errors).run(callback)
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 904, in run
    if self.complete() is True:
  File "/usr/lib/python2.7/dist-packages/deployer/env/watchers.py", line 77, in complete
    on_errors(new_errors)
  File "/usr/lib/python2.7/dist-packages/deployer/env/watchers.py", line 98, in callback
    raise ErrorExit()
ErrorExit
==================================================================

From juju_status.yaml:
==================================================================
machines:
  '0':
    agent-state: started
    agent-version: 1.22-beta2
    dns-name: hayward-12.oil
    hardware: arch=amd64 cpu-cores=8 mem=16384M tags=oil-slave-2,hardware-sm15k,hw-staging,bug1352513
    instance-id: /MAAS/api/1.0/nodes/node-a2343d38-c4cd-11e3-824b-00163efc5068/
    series: trusty
    state-server-member-status: has-vote
  '1':
    agent-state: started
    agent-version: 1.22-beta2
    containers:
      1/lxc/0:
        agent-state-info: 'failed to retrieve the template to clone: lxc container
          creation failed: error executing "lxc-create": + ''['' amd64 = i686 '']'';
          + ''['' 0 = 0 '']''; + case "$hostarch:$arch" in; + :; + ''['' tryreleased
          ''!='' daily -a tryreleased ''!='' released -a tryreleased ''!='' tryreleased
          '']''; + ''['' -z /var/lib/lxc/juju-trusty-lxc-template '']''; ++ id -u;
          + ''['' 0 ''!='' 0 '']''; + config=/var/lib/lxc/juju-trusty-lxc-template/config;
          + ''['' -z /var/lib/lxc/juju-trusty-lxc-template/rootfs '']''; + type ubuntu-cloudimg-query;
          ubuntu-cloudimg-query is /usr/bin/ubuntu-cloudimg-query; + type wget; wget
          is /tmp/wget450531556/wget; + cache=/var/cache/lxc/cloud-trusty; + ''[''
          0 -eq 1 '']''; + mkdir -p /var/cache/lxc/cloud-trusty; + ''['' tryreleased
          = tryreleased '']''; + stream=released; + ubuntu-cloudimg-query trusty released
          amd64; + ''['' -n https://10.245.0.201:17070/environment/5bc8bc49-3ca9-42f4-82e8-441128568600/images/lxc/trusty/amd64/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz
          '']''; + url2=https://10.245.0.201:17070/environment/5bc8bc49-3ca9-42f4-82e8-441128568600/images/lxc/trusty/amd64/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz;
          ++ basename https://10.245.0.201:17070/environment/5bc8bc49-3ca9-42f4-82e8-441128568600/images/lxc/trusty/amd64/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz;
          + filename=ubuntu-14.04-server-cloudimg-amd64-root.tar.gz; + ''['' -n https://10.245.0.201:17070/environment/5bc8bc49-3ca9-42f4-82e8-441128568600/images/lxc/trusty/amd64/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz
          '']''; + do_extract_rootfs; + cd /var/cache/lxc/cloud-trusty; + ''['' 0
          -eq 1 '']''; + trap wgetcleanup EXIT SIGHUP SIGINT SIGTERM; + ''['' ''!''
          -f ubuntu-14.04-server-cloudimg-amd64-root.tar.gz '']''; + wget https://10.245.0.201:17070/environment/5bc8bc49-3ca9-42f4-82e8-441128568600/images/lxc/trusty/amd64/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz;
          --2015-02-03 12:58:37-- https://10.245.0.201:17070/environment/5bc8bc49-3ca9-42f4-82e8-441128568600/images/lxc/trusty/amd64/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz;
          Connecting to 10.245.0.201:17070... connected.; HTTP request sent, awaiting
          response... 500 Internal Server Error; 2015-02-03 12:58:37 ERROR 500: Internal
          Server Error.; + build_root_tgz ubuntu-14.04-server-cloudimg-amd64-root.tar.gz;
          + url=ubuntu-14.04-server-cloudimg-amd64-root.tar.gz; + filename=; ++ mktemp
          -d -p .; + xdir=./tmp.uG5Tb52zkj; ++ basename ubuntu-14.04-server-cloudimg-amd64-root.tar.gz;
          + tarname=ubuntu-14.04-server-cloudimg-amd64-root.tar.gz; + imgname=''trusty-*-cloudimg-amd64.img'';
          + trap buildcleanup EXIT SIGHUP SIGINT SIGTERM; + ''['' 0 -eq 1 -o ''!''
          -f /var/cache/lxc/cloud-trusty/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz
          '']''; + rm -f ubuntu-14.04-server-cloudimg-amd64-root.tar.gz; + echo ''Downloading
          cloud image from ubuntu-14.04-server-cloudimg-amd64-root.tar.gz''; Downloading
          cloud image from ubuntu-14.04-server-cloudimg-amd64-root.tar.gz; + wget
          ubuntu-14.04-server-cloudimg-amd64-root.tar.gz; --2015-02-03 12:58:37-- http://ubuntu-14.04-server-cloudimg-amd64-root.tar.gz/;
          Resolving ubuntu-14.04-server-cloudimg-amd64-root.tar.gz (ubuntu-14.04-server-cloudimg-amd64-root.tar.gz)...
          failed: Name or service not known.; wget: unable to resolve host address
          ''ubuntu-14.04-server-cloudimg-amd64-root.tar.gz''; + echo ''Couldn''\''''t
          find cloud image ubuntu-14.04-server-cloudimg-amd64-root.tar.gz.''; Couldn''t
          find cloud image ubuntu-14.04-server-cloudimg-amd64-root.tar.gz.; + exit
          1; + buildcleanup; + cd /var/lib/lxc/juju-trusty-lxc-template/rootfs; +
          umount -l /var/cache/lxc/cloud-trusty/./tmp.uG5Tb52zkj; umount: /var/cache/lxc/cloud-trusty/./tmp.uG5Tb52zkj:
          not mounted; + true; + rm -rf /var/cache/lxc/cloud-trusty; lxc_container:
          lxccontainer.c: create_run_template: 1125 container creation template for
          juju-trusty-lxc-template failed; lxc_container: lxc_create.c: main: 271
          Error creating container juju-trusty-lxc-template'
        instance-id: pending
        series: trusty
      1/lxc/1:
        agent-state-info: 'failed to retrieve the template to clone: lxc container
          creation failed: error executing "lxc-create": + ''['' amd64 = i686 '']'';
          + ''['' 0 = 0 '']''; + case "$hostarch:$arch" in; + :; + ''['' tryreleased
          ''!='' daily -a tryreleased ''!='' released -a tryreleased ''!='' tryreleased
          '']''; + ''['' -z /var/lib/lxc/juju-trusty-lxc-template '']''; ++ id -u;
          + ''['' 0 ''!='' 0 '']''; + config=/var/lib/lxc/juju-trusty-lxc-template/config;
          + ''['' -z /var/lib/lxc/juju-trusty-lxc-template/rootfs '']''; + type ubuntu-cloudimg-query;
          ubuntu-cloudimg-query is /usr/bin/ubuntu-cloudimg-query; + type wget; wget
          is /tmp/wget495971571/wget; + cache=/var/cache/lxc/cloud-trusty; + ''[''
          0 -eq 1 '']''; + mkdir -p /var/cache/lxc/cloud-trusty; + ''['' tryreleased
          = tryreleased '']''; + stream=released; + ubuntu-cloudimg-query trusty released
          amd64; + ''['' -n https://10.245.0.201:17070/environment/5bc8bc49-3ca9-42f4-82e8-441128568600/images/lxc/trusty/amd64/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz
          '']''; + url2=https://10.245.0.201:17070/environment/5bc8bc49-3ca9-42f4-82e8-441128568600/images/lxc/trusty/amd64/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz;
          ++ basename https://10.245.0.201:17070/environment/5bc8bc49-3ca9-42f4-82e8-441128568600/images/lxc/trusty/amd64/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz;
          + filename=ubuntu-14.04-server-cloudimg-amd64-root.tar.gz; + ''['' -n https://10.245.0.201:17070/environment/5bc8bc49-3ca9-42f4-82e8-441128568600/images/lxc/trusty/amd64/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz
          '']''; + do_extract_rootfs; + cd /var/cache/lxc/cloud-trusty; + ''['' 0
          -eq 1 '']''; + trap wgetcleanup EXIT SIGHUP SIGINT SIGTERM; + ''['' ''!''
          -f ubuntu-14.04-server-cloudimg-amd64-root.tar.gz '']''; + wget https://10.245.0.201:17070/environment/5bc8bc49-3ca9-42f4-82e8-441128568600/images/lxc/trusty/amd64/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz;
          --2015-02-03 12:58:40-- https://10.245.0.201:17070/environment/5bc8bc49-3ca9-42f4-82e8-441128568600/images/lxc/trusty/amd64/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz;
          Connecting to 10.245.0.201:17070... connected.; HTTP request sent, awaiting
          response... 500 Internal Server Error; 2015-02-03 12:58:40 ERROR 500: Internal
          Server Error.; + build_root_tgz ubuntu-14.04-server-cloudimg-amd64-root.tar.gz;
          + url=ubuntu-14.04-server-cloudimg-amd64-root.tar.gz; + filename=; ++ mktemp
          -d -p .; + xdir=./tmp.opNVN6hsaa; ++ basename ubuntu-14.04-server-cloudimg-amd64-root.tar.gz;
          + tarname=ubuntu-14.04-server-cloudimg-amd64-root.tar.gz; + imgname=''trusty-*-cloudimg-amd64.img'';
          + trap buildcleanup EXIT SIGHUP SIGINT SIGTERM; + ''['' 0 -eq 1 -o ''!''
          -f /var/cache/lxc/cloud-trusty/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz
          '']''; + rm -f ubuntu-14.04-server-cloudimg-amd64-root.tar.gz; + echo ''Downloading
          cloud image from ubuntu-14.04-server-cloudimg-amd64-root.tar.gz''; Downloading
          cloud image from ubuntu-14.04-server-cloudimg-amd64-root.tar.gz; + wget
          ubuntu-14.04-server-cloudimg-amd64-root.tar.gz; --2015-02-03 12:58:40-- http://ubuntu-14.04-server-cloudimg-amd64-root.tar.gz/;
          Resolving ubuntu-14.04-server-cloudimg-amd64-root.tar.gz (ubuntu-14.04-server-cloudimg-amd64-root.tar.gz)...
          failed: Name or service not known.; wget: unable to resolve host address
          ''ubuntu-14.04-server-cloudimg-amd64-root.tar.gz''; + echo ''Couldn''\''''t
          find cloud image ubuntu-14.04-server-cloudimg-amd64-root.tar.gz.''; Couldn''t
          find cloud image ubuntu-14.04-server-cloudimg-amd64-root.tar.gz.; + exit
          1; + buildcleanup; + cd /var/lib/lxc/juju-trusty-lxc-template/rootfs; +
          umount -l /var/cache/lxc/cloud-trusty/./tmp.opNVN6hsaa; umount: /var/cache/lxc/cloud-trusty/./tmp.opNVN6hsaa:
          not mounted; + true; + rm -rf /var/cache/lxc/cloud-trusty; lxc_container:
          lxccontainer.c: create_run_template: 1125 container creation template for
          juju-trusty-lxc-template failed; lxc_container: lxc_create.c: main: 271
          Error creating container juju-trusty-lxc-template'
        instance-id: pending
        series: trusty
    dns-name: hoskins.oil
    hardware: arch=amd64 cpu-cores=12 mem=32768M tags=hw-staging,disable_biosdevname,hardware-hp-proliant-DL360E-G8,hw-staging-greg
    instance-id: /MAAS/api/1.0/nodes/node-a51d4998-f8be-11e3-a5f6-00163eca07b6/
    series: trusty

==================================================================

Larry Michel (lmic) on 2015-02-03
tags: added: oil
Curtis Hovey (sinzui) on 2015-02-03
Changed in juju-core:
status: New → Triaged
importance: Undecided → High
milestone: none → 1.23
tags: added: lxc
Ian Booth (wallyworld) wrote :

Can we please get the Juju log files uploaded

Jason Hobbs (jason-hobbs) wrote :

The problem here is the bootstrap node doesn't have cloud-image-utils installed on it. That's because of bug 1407699, where installing cloud-image-utils on precise causes cloud-init to be uninstalled. It looks like the upstream packaging for cloud-init needs to be fixed so that juju can install cloud-image-utils.

Jason Hobbs (jason-hobbs) wrote :

After the packaging for cloud-init is fixed we'd need knew cloud images to be generated with the new cloud-init package.

Ian Booth (wallyworld) wrote :

For beta 3 release, we are going to disable lxc image caching. We will fix upstream packaging, and re-enable in 1.22 beta4.

Dimiter Naydenov (dimitern) wrote :

A fix for this landed as part of the fix for bug 1423036, which is caused by the same underlying issue.

Changed in juju-core:
assignee: nobody → Ian Booth (wallyworld)
status: Triaged → Fix Committed
Curtis Hovey (sinzui) on 2015-03-09
Changed in juju-core:
status: Fix Committed → Fix Released
Curtis Hovey (sinzui) on 2015-03-09
Changed in juju-core:
milestone: 1.23 → 1.23-beta1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.
Subscribing...

Other bug subscribers