juju-core

failure to retrieve the template to clone: lxc container with 1.22 beta2

Bug #1417594 reported by Larry Michel
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
juju-core
Fix Released
High
Ian Booth
juju-core 1.23-beta1
1.22
Fix Released
High
Ian Booth
juju-core 1.22-beta3

Bug Description

We're seeing this issue on all the nodes when using 1.22 beta2:

From console:
==================================================================
2015-02-03 12:49:29,392 [DEBUG] deployer.import: Waiting for units before adding relations
2015-02-03 12:49:45,539 [DEBUG] deployer.env: Delta machine: 5 change:pending
2015-02-03 12:50:05,541 [DEBUG] deployer.env: Delta machine: 4 change:pending
2015-02-03 12:50:05,544 [DEBUG] deployer.env: Delta machine: 5 change:pending
2015-02-03 12:53:25,566 [DEBUG] deployer.env: Delta machine: 2 change:pending
2015-02-03 12:53:25,570 [DEBUG] deployer.env: Delta machine: 2 change:started
2015-02-03 12:53:25,570 [DEBUG] deployer.env: Delta machine: 3 change:pending
2015-02-03 12:53:25,570 [DEBUG] deployer.env: Delta unit: nova-cloud-controller/0 change:allocating
2015-02-03 12:53:30,569 [DEBUG] deployer.env: Delta machine: 3 change:pending
2015-02-03 12:53:30,573 [DEBUG] deployer.env: Delta machine: 3 change:started
2015-02-03 12:53:30,576 [DEBUG] deployer.env: Delta unit: nova-cloud-controller/0 change:installing
2015-02-03 12:53:30,581 [DEBUG] deployer.env: Delta unit: swift-storage/0 change:installing
2015-02-03 12:53:30,581 [DEBUG] deployer.env: Delta unit: nova-vmware/0 change:installing
2015-02-03 12:53:35,581 [DEBUG] deployer.env: Delta machine: 2/lxc/0 change:error
2015-02-03 12:53:40,582 [DEBUG] deployer.env: Delta machine: 3/lxc/0 change:error
2015-02-03 12:53:40,585 [DEBUG] deployer.env: Delta machine: 3/lxc/1 change:error
2015-02-03 12:54:40,597 [DEBUG] deployer.env: Delta unit: nova-cloud-controller/0 change:installing
2015-02-03 12:58:05,621 [DEBUG] deployer.env: Delta machine: 1 change:pending
2015-02-03 12:58:05,624 [DEBUG] deployer.env: Delta machine: 1 change:started
2015-02-03 12:58:05,624 [DEBUG] deployer.env: Delta unit: cinder/0 change:allocating
2015-02-03 12:58:10,623 [DEBUG] deployer.env: Delta unit: cinder/0 change:installing
2015-02-03 12:58:10,625 [DEBUG] deployer.env: Delta unit: cinder/0 change:installing
2015-02-03 12:58:40,628 [DEBUG] deployer.env: Delta machine: 1/lxc/0 change:error
2015-02-03 12:58:45,629 [DEBUG] deployer.env: Delta machine: 1/lxc/1 change:error
2015-02-03 12:59:50,637 [DEBUG] deployer.env: Delta machine: 5 change:pending
2015-02-03 12:59:50,641 [DEBUG] deployer.env: Delta machine: 5 change:started
2015-02-03 12:59:50,641 [DEBUG] deployer.env: Delta unit: swift-storage/2 change:installing
2015-02-03 13:00:10,642 [DEBUG] deployer.env: Delta unit: nova-vmware/0 change:error
2015-02-03 13:00:10,643 [ERROR] deployer.env: The following units had errors:
   unit: nova-vmware/0: machine: 3 agent-state: error details: hook failed: "install"
2015-02-03 13:00:10,646 [ERROR] oil_ci.deploy.oil_deployer: Deployment failed:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/oil_ci/deploy/oil_deployer.py", line 140, in deploy
    debug=True, verbose=True)
  File "/usr/lib/python2.7/dist-packages/oil_ci/juju/juju_deployer.py", line 85, in run_deployer
    importer.Importer(env, deploy, options=opts).run()
  File "/usr/lib/python2.7/dist-packages/deployer/action/importer.py", line 206, in run
    self.wait_for_units(ignore_errors=ignore_errors)
  File "/usr/lib/python2.7/dist-packages/deployer/action/importer.py", line 181, in wait_for_units
    services=self.deployment.get_service_names(), on_errors=on_errors)
  File "/usr/lib/python2.7/dist-packages/deployer/env/go.py", line 227, in wait_for_units
    services=services, on_errors=on_errors).run(callback)
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 904, in run
    if self.complete() is True:
  File "/usr/lib/python2.7/dist-packages/deployer/env/watchers.py", line 77, in complete
    on_errors(new_errors)
  File "/usr/lib/python2.7/dist-packages/deployer/env/watchers.py", line 98, in callback
    raise ErrorExit()
ErrorExit
==================================================================

From juju_status.yaml:
==================================================================
machines:
  '0':
    agent-state: started
    agent-version: 1.22-beta2
    dns-name: hayward-12.oil
    hardware: arch=amd64 cpu-cores=8 mem=16384M tags=oil-slave-2,hardware-sm15k,hw-staging,bug1352513
    instance-id: /MAAS/api/1.0/nodes/node-a2343d38-c4cd-11e3-824b-00163efc5068/
    series: trusty
    state-server-member-status: has-vote
  '1':
    agent-state: started
    agent-version: 1.22-beta2
    containers:
      1/lxc/0:
        agent-state-info: 'failed to retrieve the template to clone: lxc container
          creation failed: error executing "lxc-create": + ''['' amd64 = i686 '']'';
          + ''['' 0 = 0 '']''; + case "$hostarch:$arch" in; + :; + ''['' tryreleased
          ''!='' daily -a tryreleased ''!='' released -a tryreleased ''!='' tryreleased
          '']''; + ''['' -z /var/lib/lxc/juju-trusty-lxc-template '']''; ++ id -u;
          + ''['' 0 ''!='' 0 '']''; + config=/var/lib/lxc/juju-trusty-lxc-template/config;
          + ''['' -z /var/lib/lxc/juju-trusty-lxc-template/rootfs '']''; + type ubuntu-cloudimg-query;
          ubuntu-cloudimg-query is /usr/bin/ubuntu-cloudimg-query; + type wget; wget
          is /tmp/wget450531556/wget; + cache=/var/cache/lxc/cloud-trusty; + ''[''
          0 -eq 1 '']''; + mkdir -p /var/cache/lxc/cloud-trusty; + ''['' tryreleased
          = tryreleased '']''; + stream=released; + ubuntu-cloudimg-query trusty released
          amd64; + ''['' -n https://10.245.0.201:17070/environment/5bc8bc49-3ca9-42f4-82e8-441128568600/images/lxc/trusty/amd64/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz
          '']''; + url2=https://10.245.0.201:17070/environment/5bc8bc49-3ca9-42f4-82e8-441128568600/images/lxc/trusty/amd64/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz;
          ++ basename https://10.245.0.201:17070/environment/5bc8bc49-3ca9-42f4-82e8-441128568600/images/lxc/trusty/amd64/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz;
          + filename=ubuntu-14.04-server-cloudimg-amd64-root.tar.gz; + ''['' -n https://10.245.0.201:17070/environment/5bc8bc49-3ca9-42f4-82e8-441128568600/images/lxc/trusty/amd64/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz
          '']''; + do_extract_rootfs; + cd /var/cache/lxc/cloud-trusty; + ''['' 0
          -eq 1 '']''; + trap wgetcleanup EXIT SIGHUP SIGINT SIGTERM; + ''['' ''!''
          -f ubuntu-14.04-server-cloudimg-amd64-root.tar.gz '']''; + wget https://10.245.0.201:17070/environment/5bc8bc49-3ca9-42f4-82e8-441128568600/images/lxc/trusty/amd64/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz;
          --2015-02-03 12:58:37-- https://10.245.0.201:17070/environment/5bc8bc49-3ca9-42f4-82e8-441128568600/images/lxc/trusty/amd64/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz;
          Connecting to 10.245.0.201:17070... connected.; HTTP request sent, awaiting
          response... 500 Internal Server Error; 2015-02-03 12:58:37 ERROR 500: Internal
          Server Error.; + build_root_tgz ubuntu-14.04-server-cloudimg-amd64-root.tar.gz;
          + url=ubuntu-14.04-server-cloudimg-amd64-root.tar.gz; + filename=; ++ mktemp
          -d -p .; + xdir=./tmp.uG5Tb52zkj; ++ basename ubuntu-14.04-server-cloudimg-amd64-root.tar.gz;
          + tarname=ubuntu-14.04-server-cloudimg-amd64-root.tar.gz; + imgname=''trusty-*-cloudimg-amd64.img'';
          + trap buildcleanup EXIT SIGHUP SIGINT SIGTERM; + ''['' 0 -eq 1 -o ''!''
          -f /var/cache/lxc/cloud-trusty/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz
          '']''; + rm -f ubuntu-14.04-server-cloudimg-amd64-root.tar.gz; + echo ''Downloading
          cloud image from ubuntu-14.04-server-cloudimg-amd64-root.tar.gz''; Downloading
          cloud image from ubuntu-14.04-server-cloudimg-amd64-root.tar.gz; + wget
          ubuntu-14.04-server-cloudimg-amd64-root.tar.gz; --2015-02-03 12:58:37-- http://ubuntu-14.04-server-cloudimg-amd64-root.tar.gz/;
          Resolving ubuntu-14.04-server-cloudimg-amd64-root.tar.gz (ubuntu-14.04-server-cloudimg-amd64-root.tar.gz)...
          failed: Name or service not known.; wget: unable to resolve host address
          ''ubuntu-14.04-server-cloudimg-amd64-root.tar.gz''; + echo ''Couldn''\''''t
          find cloud image ubuntu-14.04-server-cloudimg-amd64-root.tar.gz.''; Couldn''t
          find cloud image ubuntu-14.04-server-cloudimg-amd64-root.tar.gz.; + exit
          1; + buildcleanup; + cd /var/lib/lxc/juju-trusty-lxc-template/rootfs; +
          umount -l /var/cache/lxc/cloud-trusty/./tmp.uG5Tb52zkj; umount: /var/cache/lxc/cloud-trusty/./tmp.uG5Tb52zkj:
          not mounted; + true; + rm -rf /var/cache/lxc/cloud-trusty; lxc_container:
          lxccontainer.c: create_run_template: 1125 container creation template for
          juju-trusty-lxc-template failed; lxc_container: lxc_create.c: main: 271
          Error creating container juju-trusty-lxc-template'
        instance-id: pending
        series: trusty
      1/lxc/1:
        agent-state-info: 'failed to retrieve the template to clone: lxc container
          creation failed: error executing "lxc-create": + ''['' amd64 = i686 '']'';
          + ''['' 0 = 0 '']''; + case "$hostarch:$arch" in; + :; + ''['' tryreleased
          ''!='' daily -a tryreleased ''!='' released -a tryreleased ''!='' tryreleased
          '']''; + ''['' -z /var/lib/lxc/juju-trusty-lxc-template '']''; ++ id -u;
          + ''['' 0 ''!='' 0 '']''; + config=/var/lib/lxc/juju-trusty-lxc-template/config;
          + ''['' -z /var/lib/lxc/juju-trusty-lxc-template/rootfs '']''; + type ubuntu-cloudimg-query;
          ubuntu-cloudimg-query is /usr/bin/ubuntu-cloudimg-query; + type wget; wget
          is /tmp/wget495971571/wget; + cache=/var/cache/lxc/cloud-trusty; + ''[''
          0 -eq 1 '']''; + mkdir -p /var/cache/lxc/cloud-trusty; + ''['' tryreleased
          = tryreleased '']''; + stream=released; + ubuntu-cloudimg-query trusty released
          amd64; + ''['' -n https://10.245.0.201:17070/environment/5bc8bc49-3ca9-42f4-82e8-441128568600/images/lxc/trusty/amd64/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz
          '']''; + url2=https://10.245.0.201:17070/environment/5bc8bc49-3ca9-42f4-82e8-441128568600/images/lxc/trusty/amd64/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz;
          ++ basename https://10.245.0.201:17070/environment/5bc8bc49-3ca9-42f4-82e8-441128568600/images/lxc/trusty/amd64/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz;
          + filename=ubuntu-14.04-server-cloudimg-amd64-root.tar.gz; + ''['' -n https://10.245.0.201:17070/environment/5bc8bc49-3ca9-42f4-82e8-441128568600/images/lxc/trusty/amd64/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz
          '']''; + do_extract_rootfs; + cd /var/cache/lxc/cloud-trusty; + ''['' 0
          -eq 1 '']''; + trap wgetcleanup EXIT SIGHUP SIGINT SIGTERM; + ''['' ''!''
          -f ubuntu-14.04-server-cloudimg-amd64-root.tar.gz '']''; + wget https://10.245.0.201:17070/environment/5bc8bc49-3ca9-42f4-82e8-441128568600/images/lxc/trusty/amd64/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz;
          --2015-02-03 12:58:40-- https://10.245.0.201:17070/environment/5bc8bc49-3ca9-42f4-82e8-441128568600/images/lxc/trusty/amd64/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz;
          Connecting to 10.245.0.201:17070... connected.; HTTP request sent, awaiting
          response... 500 Internal Server Error; 2015-02-03 12:58:40 ERROR 500: Internal
          Server Error.; + build_root_tgz ubuntu-14.04-server-cloudimg-amd64-root.tar.gz;
          + url=ubuntu-14.04-server-cloudimg-amd64-root.tar.gz; + filename=; ++ mktemp
          -d -p .; + xdir=./tmp.opNVN6hsaa; ++ basename ubuntu-14.04-server-cloudimg-amd64-root.tar.gz;
          + tarname=ubuntu-14.04-server-cloudimg-amd64-root.tar.gz; + imgname=''trusty-*-cloudimg-amd64.img'';
          + trap buildcleanup EXIT SIGHUP SIGINT SIGTERM; + ''['' 0 -eq 1 -o ''!''
          -f /var/cache/lxc/cloud-trusty/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz
          '']''; + rm -f ubuntu-14.04-server-cloudimg-amd64-root.tar.gz; + echo ''Downloading
          cloud image from ubuntu-14.04-server-cloudimg-amd64-root.tar.gz''; Downloading
          cloud image from ubuntu-14.04-server-cloudimg-amd64-root.tar.gz; + wget
          ubuntu-14.04-server-cloudimg-amd64-root.tar.gz; --2015-02-03 12:58:40-- http://ubuntu-14.04-server-cloudimg-amd64-root.tar.gz/;
          Resolving ubuntu-14.04-server-cloudimg-amd64-root.tar.gz (ubuntu-14.04-server-cloudimg-amd64-root.tar.gz)...
          failed: Name or service not known.; wget: unable to resolve host address
          ''ubuntu-14.04-server-cloudimg-amd64-root.tar.gz''; + echo ''Couldn''\''''t
          find cloud image ubuntu-14.04-server-cloudimg-amd64-root.tar.gz.''; Couldn''t
          find cloud image ubuntu-14.04-server-cloudimg-amd64-root.tar.gz.; + exit
          1; + buildcleanup; + cd /var/lib/lxc/juju-trusty-lxc-template/rootfs; +
          umount -l /var/cache/lxc/cloud-trusty/./tmp.opNVN6hsaa; umount: /var/cache/lxc/cloud-trusty/./tmp.opNVN6hsaa:
          not mounted; + true; + rm -rf /var/cache/lxc/cloud-trusty; lxc_container:
          lxccontainer.c: create_run_template: 1125 container creation template for
          juju-trusty-lxc-template failed; lxc_container: lxc_create.c: main: 271
          Error creating container juju-trusty-lxc-template'
        instance-id: pending
        series: trusty
    dns-name: hoskins.oil
    hardware: arch=amd64 cpu-cores=12 mem=32768M tags=hw-staging,disable_biosdevname,hardware-hp-proliant-DL360E-G8,hw-staging-greg
    instance-id: /MAAS/api/1.0/nodes/node-a51d4998-f8be-11e3-a5f6-00163eca07b6/
    series: trusty

==================================================================

Tags: lxc oil
Larry Michel (lmic)
tags: added: oil
Curtis Hovey (sinzui)
Changed in juju-core:
status: New → Triaged
importance: Undecided → High
milestone: none → 1.23
tags: added: lxc
Revision history for this message
Ian Booth (wallyworld) wrote :

Can we please get the Juju log files uploaded

Revision history for this message
Jason Hobbs (jason-hobbs) wrote :

The problem here is the bootstrap node doesn't have cloud-image-utils installed on it. That's because of bug 1407699, where installing cloud-image-utils on precise causes cloud-init to be uninstalled. It looks like the upstream packaging for cloud-init needs to be fixed so that juju can install cloud-image-utils.

Revision history for this message
Jason Hobbs (jason-hobbs) wrote :

After the packaging for cloud-init is fixed we'd need knew cloud images to be generated with the new cloud-init package.

Revision history for this message
Ian Booth (wallyworld) wrote :

For beta 3 release, we are going to disable lxc image caching. We will fix upstream packaging, and re-enable in 1.22 beta4.

Revision history for this message
Dimiter Naydenov (dimitern) wrote :

A fix for this landed as part of the fix for bug 1423036, which is caused by the same underlying issue.

Changed in juju-core:
assignee: nobody → Ian Booth (wallyworld)
status: Triaged → Fix Committed
Curtis Hovey (sinzui)
Changed in juju-core:
status: Fix Committed → Fix Released
Curtis Hovey (sinzui)
Changed in juju-core:
milestone: 1.23 → 1.23-beta1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.