Deployment on 98 nodes with Ceilo on CentOS failed due to wrong ceph roles layout

[Sergey Galkin]

astute_sha: c15623d05ccdf7ac10873e7a90df954de8726280
auth_required: true
build_id: 2014-11-26_13-35-04
build_number: '10'
feature_groups:
- mirantis
fuellib_sha: 25eb629f3c6a6ff41cf187e260fe4ff456cfc4e4
fuelmain_sha: 465afb6479a0b3c677040fb978cc109dcf62f774
nailgun_sha: bf9ddb9f9d5dbb09c4b50201ce176635791d7d3e
ostf_sha: a35f516f1606b0d03d51ff63bfe3fbe23de4b622
production: docker
release: '6.0'

Steps to reproduce
1. Start deploy cluster with 96 compute nodes, 3 controlles in HA on CentOS with cinder, neutron gre, ceilometer.
Deployment has failed with error:
 Deployment has failed. Method deploy. Upload cirros "TestVM" image failed.
Inspect Astute logs for the details

In the astute logs

2014-11-28 15:42:15 ERR
[417] Error running RPC method deploy: Upload cirros "TestVM" image failed, trace:
["/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/post_deployment_actions/upload_cirros_image.rb:109:in `raise_cirros_error'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/post_deployment_actions/upload_cirros_image.rb:92:in `process'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/deploy_actions.rb:25:in `block in process'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/deploy_actions.rb:25:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/deploy_actions.rb:25:in `process'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/deployment_engine.rb:95:in `deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/orchestrator.rb:264:in `deploy_cluster'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/orchestrator.rb:35:in `deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/dispatcher.rb:59:in `deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:142:in `dispatch_message'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:103:in `block in dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:64:in `call'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:64:in `block in each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:101:in `each_with_index'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:101:in `dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:85:in `block in perform_main_job'"]
2014-11-28 15:42:15 ERR
[417] d95e44a4-fc76-4389-a940-64742b97ab7b: Upload cirros "TestVM" image failed
2014-11-28 15:42:15 ERR
[417] d95e44a4-fc76-4389-a940-64742b97ab7b: cmd: . /root/openrc && /usr/bin/glance image-create --name 'TestVM' --is-public true --container-format='bare' --disk-format='qcow2' --min-ram=64 --property murano_image_info='{"title": "Murano Demo", "type": "cirros.demo"}' --file '/opt/vm/cirros-x86_64-disk.img'
                                               mcollective error: d95e44a4-fc76-4389-a940-64742b97ab7b: MCollective agents '32' didn't respond within the allotted time.
2014-11-28 15:42:15 ERR
[417] MCollective agents '32' didn't respond within the allotted time.

[Bogdan Dobrelya (bogdando)]

Please link a logs

[Egor Kotko (ykotko)]

Have similar on Ubuntu:

http://paste.openstack.org/show/142420/

{"build_id": "2014-11-27_23-41-13", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "45", "auth_required": true, "api": "1.0", "nailgun_sha": "500e36d08a45dbb389bf2bd97673d9bff48ee84d", "production": "docker", "fuelmain_sha": "51e66db7750e9c856ba128f35cfb6724895bf479", "astute_sha": "ef8aa0fd0e3ce20709612906f1f0551b5682a6ce", "feature_groups": ["mirantis"], "release": "5.1.1", "release_versions": {"2014.1.3-5.1.1": {"VERSION": {"build_id": "2014-11-27_23-41-13", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "45", "api": "1.0", "nailgun_sha": "500e36d08a45dbb389bf2bd97673d9bff48ee84d", "production": "docker", "fuelmain_sha": "51e66db7750e9c856ba128f35cfb6724895bf479", "astute_sha": "ef8aa0fd0e3ce20709612906f1f0551b5682a6ce", "feature_groups": ["mirantis"], "release": "5.1.1", "fuellib_sha": "15a387462f7be50c4f87ad986d0c81535025c125"}}}, "fuellib_sha": "15a387462f7be50c4f87ad986d0c81535025c125"}

[Nastya Urlapova (aurlapova)]

Guys, it is really confirmed for 5.1.1, as I remember we certified 5.1.1 only on 20 nodes.

[Bogdan Dobrelya (bogdando)]

We cannot address high bugs in 5.1.1 anyway, due to HCF

[Bogdan Dobrelya (bogdando)]

@Nastya, but you're right we do not support 100 nodes deployment in 5.1.x

[Bogdan Dobrelya (bogdando)]

Well, nodes *were* provisioned, though
2014-11-28T16:27:46 info: [421] All nodes are provisioned

will continue investigate the logs in #2

[Bogdan Dobrelya (bogdando)]

The deployment logs from astute are ok, I can see also glance report about successful image creation, but it came a bit later than astute expected: http://paste.openstack.org/show/142646/

[Vladimir Sharshov (vsharshov)]

Yes, Bogdan is right. Glance image should upload not more than 60 seconds. Glance image size ~ 16 mb. If this operation took more then 60 seconds - we have cinder performance degradation.

This bug similar with https://bugs.launchpad.net/fuel/+bug/1397254 (the same signs ).

I believe that this error not in Astute and we need discovery cinder performance.

[Mike Scherbakov (mihgen)]

MOS team, please analyse openstack logs - we need to understand why it took too much to upload an image into Glance. This could be related to Glance, Keystone, Glance backend...

[Alexander Tivelkov (ativelkov)]

The message
INFO glance.registry.api.v1.images [911143ba-bf4b-4847-a852-77385abd308a 97bea994c8cd4e16b8701d1dd776272f 18d3f86359684daa94bc83fc6f8868eb - - -] Successfully created image 4dc69685-c45f-4193-bd50-6c2743c9dd4a
Does not mean that the image was indeed uploaded, it just says that the record has been created in the database (that's what the registry service does). The actual uploading is made in glance-api service, and it seems like its last log message is

2014-11-28T17:18:46.437043+00:00 debug: 2014-11-28 17:18:46.450 26587 DEBUG glance.store.rbd [9ec256a4-29b6-4c14-a4dd-11c64a5d796f 97bea994c8cd4e16b8701d1dd776272f 18d3f86359684daa94bc83fc6f8868eb - - -] creating image 4dc69685-c45f-4193-bd50-6c2743c9dd4a with order 23 and size 13167616 add /usr/lib/python2.7/dist-packages/glance/store/rbd.py:332

I.e. glance calls to ceph and attempts to put data there. And it seems like there is no response. We need somebody to look into ceph logs to find out what's going on.

[Stanislav Makar (smakar)]

astute.yaml:
volumes_ceph: true
images_ceph: true

roles:
  role: primary-controller
  name: node-1
  role: controller
  name: node-2
  role: controller
  name: node-3
  role: compute
  name: node-4
  role: compute
  name: node-5
  role: cinder
  name: node-6
  role: cinder
  name: node-7

There are node ceph-osd nodes

[Stanislav Makar (smakar)]

There are not ceph-osd nodes

[Mike Scherbakov (mihgen)]

If comment #12 goes to wrong direction, then we have to investigate more. If there were no Ceph, then there was Swift (in HA we use Swift by default for Glance backend)

[Alexander Tivelkov (ativelkov)]

Mike, according to snapshot glance is configured to use ceph: see glance-api.conf, it has "default_store = rbd"

Also, the logs state that it was indeed attempting to put the image to ceph. If there is no ceph deployed, then something is wrong with the configuration.

[Bogdan Dobrelya (bogdando)]

According to #13, the configuration looks not valid

[Bogdan Dobrelya (bogdando)]

We should check how it was possible to deploy an env with ceph for volumes and images w/o ceph-osd roles assigned

[Bogdan Dobrelya (bogdando)]

Well, all of it was related to the https://bugs.launchpad.net/fuel/+bug/1397367/comments/2 actually. The original bug does not provide the logs

[Bogdan Dobrelya (bogdando)]

I suggest to resubmit the logs from #2 and #3 to the separate bug about ceph roles layout and close this one as invalid due to no logs