[systest] HA flat add compute - step wait for ha proxy keystone backend failed

[Tatyanka]

Latest one:
Reproduced on Centos HA flat add compute
with 504 ERROR from keystone on auth
http://paste.openstack.org/show/93826/

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "5.1"
  api: "1.0"
  build_number: "424"
  build_id: "2014-08-12_02-01-17"
  astute_sha: "b52910642d6de941444901b0f20e95ebbcb2b2e9"
  fuellib_sha: "54d834afccf1f8a97e4a82d7d081e7d1a1e068a1"
  ostf_sha: "d2a894d228c1f3c22595a77f04b1e00d09d8e463"
  nailgun_sha: "c7e00e5a00499d9f8dec608541dad1d745a8dd2e"
  fuelmain_sha: "9d4463400b4924159c978af43855e48bcf2a84b2"

http://jenkins-product.srt.mirantis.net:8080/view/0_master_swarm/job/master_fuelmain.system_test.centos.thread_4/131/consoleFull

P.S. Snapshot was updated with latest result. Failed node - node-1
Also /etc/init.d status says that keystone is running but
. openrc and manual exution of nova list command showw the next result
http://paste.openstack.org/show/93827/
keystone tenenat-list command result
http://paste.openstack.org/show/93828/

Previous fail:
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "5.1"
  api: "1.0"
  build_number: "389"
  build_id: "2014-08-03_02-01-14"
  astute_sha: "ce86172e77661026c91fdf1ff8066d7df1f7d89d"
  fuellib_sha: "4e3fdd75f8dabde8e5d07067545d8043a70a176b"
  ostf_sha: "a3fa823ea0e4e03beb637ae07a91adea82c33182"
  nailgun_sha: "bd0127be0061029f9f910547db5e633c82244942"
  fuelmain_sha: "e99879292cf6e96b8991300d947df76b69134bb1"
Test ha flat add compute fail on ubuntu with next error in puppet on controller node

http://paste.openstack.org/show/90540/

[Nastya Urlapova (aurlapova)]

Looks like invalid, ha_flat_add_compute passed tree time on centos|ubuntu

[Dmitry Borodaenko (angdraug)]

ISO #389 didn't include https://review.openstack.org/111716

[Tatyanka (tatyana-leontovich)]

Dmitry, Nastya I do not agree that it is invalid/duplicate. We can see that it is reproduced on CI but we fail not with 401 error we failed with 504 error (and I beleive it is not client error and we should to be sure that everithing fine with server) - so please, take a look at the updated description and snapshots

[Bogdan Dobrelya (bogdando)]

2014-08-03T18:25:24.042159+01:00 info: (/Stage[main]/Osnailyfacter::Cluster_ha/Exec[wait-for-haproxy-keystone-backend]) Starting to evaluate the resource
...
2014-08-03T18:38:03.047263+01:00 notice: (/Stage[main]/Openstack::Ha::Keystone/Openstack::Ha::Haproxy_service[keystone-1]/Haproxy::Listen[keystone-1]/Haproxy::Service[keystone-1]/Concat[/etc/haproxy/conf.d/020-keystone-1.cfg]/Exec[concat_/etc/haproxy/conf.d/020-keystone-1.cfg]) Triggered 'refresh' from 4 events

rc: haproxy backend service triggered after its evaluation.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/113870

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/113870
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=bd89083f75b37c1119b8eb7e6e12103d4c68b6aa
Submitter: Jenkins
Branch: master

commit bd89083f75b37c1119b8eb7e6e12103d4c68b6aa
Author: Bogdan Dobrelya <email address hidden>
Date: Wed Aug 13 15:52:18 2014 +0300

    Fix haproxy backend checks dependencies

    Openstack::Ha::Haproxy_service['keystone*']
    should always be evaluated *before* any
    Exec['wait-for-keystone*'] backend checks.

    Closes-bug: #1352964

    Change-Id: I579f33d673925bea27ceab48f3380ed1dda40170
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/114508

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/114508
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=70f0f0f1ff0f6547b1d70e1206d9cbacf3d704d3
Submitter: Jenkins
Branch: stable/5.0

commit 70f0f0f1ff0f6547b1d70e1206d9cbacf3d704d3
Author: Bogdan Dobrelya <email address hidden>
Date: Wed Aug 13 15:52:18 2014 +0300

    Fix haproxy backend checks dependencies

    Openstack::Ha::Haproxy_service['keystone*']
    should always be evaluated *before* any
    Exec['wait-for-keystone*'] backend checks.

    Closes-bug: 1352964

    Change-Id: I579f33d673925bea27ceab48f3380ed1dda40170
    Signed-off-by: Bogdan Dobrelya <email address hidden>