[bvt] Deploy failed with heat-keystone-setup-domain returned 1 instead of one of [0]

[Tatyanka]

Deploy cluster in ha mode with 1 controller and Neutron GRE
OS: Ubuntu
Scenario:
1. Create cluster
2. Add 1 node with controller role
3. Add 2 nodes with compute role
4. Deploy the cluster
5. Run network verification
6. Run OSTF

Actual:
Deployment is failed with:
node-1 2015-07-28T05:10:43.626142 err: heat-keystone-setup-domain returned 1 instead of one of [0]
node-1 2015-07-28T05:10:43.626368 err: /usr/lib/ruby/vendor_ruby/puppet/util/errors.rb:97:in `fail'
node-1 2015-07-28T05:10:43.627431 err: /usr/lib/ruby/vendor_ruby/puppet/type/exec.rb:142:in `sync'
node-1 2015-07-28T05:10:43.627431 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:191:in `sync'
node-1 2015-07-28T05:10:43.627431 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:128:in `sync_if_needed'
node-1 2015-07-28T05:10:43.627431 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:89:in `block in perform_changes'
node-1 2015-07-28T05:10:43.627431 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:88:in `each'
node-1 2015-07-28T05:10:43.627431 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:88:in `perform_changes'
node-1 2015-07-28T05:10:43.627431 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:20:in `evaluate'
node-1 2015-07-28T05:10:43.627431 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:174:in `apply'
node-1 2015-07-28T05:10:43.627431 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:187:in `eval_resource'
node-1 2015-07-28T05:10:43.627431 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:117:in `call'
node-1 2015-07-28T05:10:43.627431 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:117:in `block (2 levels) in evaluate'
node-1 2015-07-28T05:10:43.627431 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:327:in `block in thinmark'
node-1 2015-07-28T05:10:43.627431 err: /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
node-1 2015-07-28T05:10:43.627431 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:326:in `thinmark'
node-1 2015-07-28T05:10:43.627431 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:117:in `block in evaluate'
node-1 2015-07-28T05:10:43.627431 err: /usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in `traverse'
node-1 2015-07-28T05:10:43.627431 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:108:in `evaluate'
node-1 2015-07-28T05:10:43.627431 err: /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:164:in `block in apply'
node-1 2015-07-28T05:10:43.627431 err: /usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
node-1 2015-07-28T05:10:43.627431 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:108:in `as_logging_destination'
node-1 2015-07-28T05:10:43.627431 err: /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:163:in `apply'
node-1 2015-07-28T05:10:43.627431 err: /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:125:in `block in apply_catalog'
node-1 2015-07-28T05:10:43.627431 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
node-1 2015-07-28T05:10:43.627671 err: /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
node-1 2015-07-28T05:10:43.627671 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
node-1 2015-07-28T05:10:43.627671 err: /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:124:in `apply_catalog'
node-1 2015-07-28T05:10:43.627671 err: /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:192:in `run'
node-1 2015-07-28T05:10:43.627671 err: /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:268:in `apply_catalog'
node-1 2015-07-28T05:10:43.627671 err: /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:218:in `main'
node-1 2015-07-28T05:10:43.627671 err: /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:146:in `run_command'
node-1 2015-07-28T05:10:43.627671 err: /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `block (2 levels) in run'
node-1 2015-07-28T05:10:43.627671 err: /usr/lib/ruby/vendor_ruby/puppet/application.rb:470:in `plugin_hook'
node-1 2015-07-28T05:10:43.627671 err: /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `block in run'
node-1 2015-07-28T05:10:43.627671 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:478:in `exit_on_fail'
node-1 2015-07-28T05:10:43.627671 err: /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `run'
node-1 2015-07-28T05:10:43.627671 err: /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:137:in `run'
node-1 2015-07-28T05:10:43.627671 err: /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:91:in `execute'
node-1 2015-07-28T05:10:43.627671 err: /usr/bin/puppet:4:in `<main>'
node-1 2015-07-28T05:10:43.628355 err: (/Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns) change from notrun to 0 failed: heat-keystone-setup-domain returned 1 instead of one of [0]

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "102"
  build_id: "2015-07-28_03-12-37"
  nailgun_sha: "fd7392bb8d9d640631d9805bf1fbe3124e98db04"
  python-fuelclient_sha: "c2f8b78686de9dceca8bb9c3f655a7fbbd1375a4"
  fuel-agent_sha: "2a65f11c10b0aeb5184247635a19740fc3edde21"
  astute_sha: "34e0493afa22999c4a07d3198ceb945116ab7932"
  fuel-library_sha: "0c29bbc15cf981a4d7d2c6ef7d9c25ba9f828873"
  fuel-ostf_sha: "9e32b35e9a774eec41b86cd5a6c63b71a3efa4bd"
  fuelmain_sha: "921918a3bd3d278431f35ad917989e46b0c24100"

[Vladimir Kuklin (vkuklin)]

Regression was introduced by this bugfix https://review.openstack.org/#/c/198655/8

After apache support for keystone was merged, each restart of apache (e.g. by horizon deployment) makes keystone unavailable on this node for more than 5 minutes.

As a temporary solution we can add waiting for keystone haproxy backend

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/206647

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/206647
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=42ebca0f03cd6b7ec91bea658337a619b5b35287
Submitter: Jenkins
Branch: master

commit 42ebca0f03cd6b7ec91bea658337a619b5b35287
Author: Dmitry Ilyin <email address hidden>
Date: Tue Jul 28 21:25:35 2015 +0300

    Add haproxy_status for keystone to the heat task

    * Wait for keystone to become online before working with it

    Change-Id: I25d837cad4b6af6bbdb2b0c137a0555c0ddaa562
    Closes-Bug: 1478842

[Tatyanka (tatyana-leontovich)]

verified 129

[Egor Kotko (ykotko)]

Reproduced on:
{"build_id": "2015-08-10_17-24-26", "build_number": "156", "release_versions": {"2015.1.0-7.0": {"VERSION": {"build_id": "2015-08-10_17-24-26", "build_number": "156", "api": "1.0", "fuel-library_sha": "c4b2b9a1b464d7f538635cbfc9304a714ededc41", "nailgun_sha": "9a6ac9b08733ded67bc65345a37787886ff4249d", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "e01693992d7a0304d926b922b43f3b747c35964c", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "57145b1d8804389304cd04322ba0fb3dc9d30327", "production": "docker", "python-fuelclient_sha": "ec5c02b3848fe1d15e5b50b323c3dda030f8e1ac", "astute_sha": "e1d3a435e5df5b40cbfb1a3acf80b4176d15a2dc", "fuel-ostf_sha": "c7f745431aa3c147f2491c865e029e0ffea91c47", "release": "7.0", "fuelmain_sha": "826387d1a05a40e4849433598442024bb1e3b7c7"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "c4b2b9a1b464d7f538635cbfc9304a714ededc41", "nailgun_sha": "9a6ac9b08733ded67bc65345a37787886ff4249d", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "e01693992d7a0304d926b922b43f3b747c35964c", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "57145b1d8804389304cd04322ba0fb3dc9d30327", "production": "docker", "python-fuelclient_sha": "ec5c02b3848fe1d15e5b50b323c3dda030f8e1ac", "astute_sha": "e1d3a435e5df5b40cbfb1a3acf80b4176d15a2dc", "fuel-ostf_sha": "c7f745431aa3c147f2491c865e029e0ffea91c47", "release": "7.0", "fuelmain_sha": "826387d1a05a40e4849433598442024bb1e3b7c7"}

[Egor Kotko (ykotko)]

http://jenkins-product.srt.mirantis.net:8080/job/7.0.system_test.ubuntu.thread_7/58/testReport/%28root%29/deploy_ceph_ha_nodegroups/deploy_ceph_ha_nodegroups/?

Scenario:
            1. Revert snapshot with 2 networks sets for slaves
            2. Create cluster (HA) with Neutron VXLAN and Ceph
            3. Add 3 controller + ceph nodes from default nodegroup
            4. Add 2 compute + ceph nodes from custom nodegroup
            5. Deploy cluster
            6. Run health checks (OSTF)

[Dmitry Ilyin (idv1985)]

Now it's something new. Keystone-admin backend is up but Authorization failed.

[Mike Scherbakov (mihgen)]

Yegor, please open a new bug, and close this one back.

[Egor Kotko (ykotko)]

https://bugs.launchpad.net/fuel/+bug/1484066