Failed to upload TestVM image to Glance in post-deployment

[Vadim Rovachev]

{"build_id": "2014-06-30_03-01-14", "mirantis": "yes", "build_number": "77", "ostf_sha": "d0fe60e0eba61685008b86d101f459fc2d3bb654", "nailgun_sha": "dd7f32ab80c023a4afda70b521dd5391e5e464fd", "production": "docker", "api": "1.0", "fuelmain_sha": "d5cd1439a382b335ec9f7ac69d2ace78c9c68120", "astute_sha": "17a1dc816d6d56dda64c2db21b94581472cabefb", "release": "5.0.1", "fuellib_sha": "7f7a7b33a5711b6146e5a11811c68e91ca4761af"}

Steps to reproduce:
1. Install master fuel.
2. Create env with parameters:
  env_name=centos-neutron-gre
virt_type=kvm
config_mode=multinode
release_name=Icehouse on CentOS 6.5
net_provider=neutron
net_segment_type=gre
debug=true
auto_assign_floating_ip=false
nova_quota=true
settings:{"volumes_lvm": false, "volumes_ceph": true, "images_ceph": true, "ephemeral_ceph": false,
 "objects_ceph": true, "osd_pool_size": 3, "sahara": false, "murano": false, "ceilometer": true}
node_roles={"node_1_controller":{"manufacturer":"KVM", "roles": ["controller", "mongo"]},
 "node_2_controller":{"manufacturer":"KVM", "roles": ["ceph-osd"]},
 "node_3_controller":{"manufacturer":"KVM", "roles": ["ceph-osd"]},
 "node_4_controller":{"manufacturer":"KVM", "roles": ["ceph-osd"]},
 "node_5_compute":{"manufacturer":"Supermicro", "roles": ["compute"]}}
3. Deploy env.
Deployment has failed. Error occurred while running method 'deploy'. Inspect Orchestrator logs for the details.

[Dmitry Borodaenko (angdraug)]

According to the attached logs, ceph cluster came up just fine:

2014-06-30 17:29:46.812452 mon.0 192.168.0.4:6789/0 29 : [INF] osd.2 192.168.0.2:6800/30592 boot
2014-06-30 17:29:46.820244 mon.0 192.168.0.4:6789/0 30 : [INF] osdmap e15: 3 osds: 3 up, 3 in
2014-06-30 17:29:47.210600 mon.0 192.168.0.4:6789/0 31 : [INF] pgmap v17: 704 pgs: 704 creating; 0 bytes data, 2080 MB used, 30107 MB / 32188 MB avail
2014-06-30 17:31:49.565706 mon.0 192.168.0.4:6789/0 39 : [INF] pgmap v23: 704 pgs: 1 active, 596 peering, 107 active+degraded; 0 bytes data, 6247 MB used, 90317 MB / 96564 MB avail
...
2014-06-30 17:36:40.005780 mon.0 192.168.0.4:6789/0 147 : [INF] pgmap v131: 704 pgs: 704 active+clean; 8192 KB data, 6273 MB used, 90291 MB / 96564 MB avail; 2631B/s rd, 1503KB/s wr, 4op/s

[Dmitry Borodaenko (angdraug)]

According to this line in astute.log, deployment of all nodes completed successfully:

2014-06-30T17:29:45 info: [394] 15f63e5f-ed5b-41ad-8ddd-d706d8423b6f: Finished deployment of nodes => roles: [{"21"=>"ceph-osd"}, {"22"=>"compute"}, {"24"=>"ceph-osd"}, {"25"=>"ceph-osd"}]

What failed was CirrOS image upload:

2014-06-30T17:29:47 debug: [394] 15f63e5f-ed5b-41ad-8ddd-d706d8423b6f: MC agent 'execute_shell_command', method 'execute', results: {:sender=>"23", :statuscode=>0, :statusmsg=>"OK", :data=>{:stdout=>"ID Name Disk Format Container Format Size \n------------------------------------ ------------------------------ -------------------- -------------------- --------------\n", :exit_code=>1, :stderr=>"/usr/lib/python2.6/site-packages/eventlet/hubs/__init__.py:8: UserWarning: Module backports was already imported from /usr/lib64/python2.6/site-packages/backports/__init__.pyc, but /usr/lib/python2.6/site-packages is being added to sys.path\n import pkg_resources\n/usr/lib/python2.6/site-packages/eventlet/hubs/__init__.py:8: UserWarning: Module backports was already imported from /usr/lib64/python2.6/site-packages/backports/__init__.pyc, but /usr/lib/python2.6/site-packages is being added to sys.path\n import pkg_resources\n"}}
2014-06-30T17:29:47 debug: [394] 15f63e5f-ed5b-41ad-8ddd-d706d8423b6f: cmd: /usr/bin/glance -N http://192.168.0.4:5000/v2.0/ -T admin -I admin -K admin index && (/usr/bin/glance -N http://192.168.0.4:5000/v2.0/ -T admin -I admin -K admin index | grep TestVM)

[Dmitry Borodaenko (angdraug)]

Please try to upload the TestVM image using the same command line manually after deployment and see if it still fails with a more informative error message.

[Dmitry Borodaenko (angdraug)]

Not reproduced for 2 weeks, closing as Invalid.