Test HA scalability fails: 'Deployment has failed. Method deploy. Disabling the upload of disk image because glance was not installed properly'

[Artem Panchenko]

api: '1.0'
astute_sha: c72dac7b31646fbedbfc56a2a87676c6d5713fcf
auth_required: true
build_id: 2014-11-05_21-28-15
build_number: '79'
feature_groups:
- mirantis
fuellib_sha: 03c63d4233f6e1c182cc58213cd4542aba5b706d
fuelmain_sha: 35eed79fd6dcfc86a9206ffcd90d385ba5dcca56
nailgun_sha: 01579a53c2be841dc4cafb0be00213243dde7667
ostf_sha: 9c6fadca272427bb933bc459e14bb1bad7f614aa
production: docker
release: '6.0'

Adding of controller nodes fails - deployment finishes with error:

'Deployment has failed. Method deploy. Disabling the upload of disk image because glance was not installed properly.
Inspect Astute logs for the details'

This bug was reproduced on CI during system tests:

http://jenkins-product.srt.mirantis.net:8080/view/6.0_swarm/job/6.0_fuelmain.system_test.ubuntu.thread_4/17/?

Steps to reproduce:

 1. Create environment HA, Ubuntu, NovaFlat.
 2. Add 1 controller node.
 3. Deploy the cluster. Result - success.
 4. Add 2 controller nodes
 5. Deploy changes. Result - success.
 6. Add 2 controller nodes.
 7. Deploy changes.

Expected result:

 - cluster is successfully deployed

Actual:

 - deployment fails (see error above)

Here is the part of Astute's logs:

http://paste.openstack.org/show/130464/

and output of 'pcs status' command on controllers:

http://paste.openstack.org/show/130465/

As you can see after 3rd deployment (step 7) 2 controllers were marked as 'error' (nodes which were added on step 4), but new 2 controllers were successfully deployed. Puppet logs on failed nodes (node-2, node-3) contain the following errors:

http://paste.openstack.org/show/130476/

Also, I tried to execute 'glance image-list' command few times and it randomly returned errors:

http://paste.openstack.org/show/130478/

Diagnostic snapshot is attached.

[Andrey Sledzinskiy (asledzinskiy)]

It failed again at the same controllers (node-3, node-4) specified in desription with
2014-11-10 09:11:13 ERR

 (/Stage[corosync_setup]/Osnailyfacter::Cluster_ha::Virtual_ips/Cluster::Virtual_ips[management]/Cluster::Virtual_ip[management]/Cs_resource[vip__management]) Could not evaluate: Execution of '/usr/sbin/cibadmin --patch --sync-call --xml-text <diff crm_feature_set='3.0.7'>

Logs are attached

[Dmitry Borodaenko (angdraug)]

This bug is blocking validation of Linux kernel upgrade.

[Matthew Mosesohn (raytrac3r)]

This could be failing because of swift configuration. This was fixed recently by Andrew Woodward.

[Dmitry Borodaenko (angdraug)]

Is this bug In Progress? Is Matt's comment #5 correct?

[Matthew Mosesohn (raytrac3r)]

Artem, can you confirm this bug is reproducable after swift fixes were merged?

[Dmitry Ilyin (idv1985)]

This bug is related to our cs_resource behaving not as good as it should be. I've tried to port upstream versions but found it both too hard and in vain because upstream version is far from perfect too.

I should repair this provider somehow.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/134964
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=c045ce3078c3d5e0d5df786041875332b8f3fac2
Submitter: Jenkins
Branch: master

commit c045ce3078c3d5e0d5df786041875332b8f3fac2
Author: Dmitry Ilyin <email address hidden>
Date: Thu Nov 27 18:11:16 2014 +0300

    Fix idempotency of cs_resource

    * insync? to drop status metadata from checks
    * code cleanup
    * fix rspec for cs_resource type
    * switch location add implementation from pcs
      to cibadmin --patch to solve problems with
      cib changes not being synced to other nodes

    related-blueprint: pacemaker-improvements
    Related-Bug: 1391599
    Related-Bug: 1390480
    Related-Bug: 1396481

    Change-Id: I5410b91ea01fc8c6805de6becdf0800d0d486188
    Signed-off-by: Sergii Golovatiuk <email address hidden>