keystone-manage db_sync failed on first controller in ha mode

[Igor Shishkin]

Steps to reproduce:

3 controllers + 2 computes, Centos, Nova network without VLAN tagging.

{
"api":
"1.0",
"astute_sha":
"2d2a44d2f7c5aa6bdbcb161bf9bac1ada20a759f",
"build_id":
"2014-05-13_11-26-46",
"build_number":
"198",
"fuellib_sha":
"30fd372a822699b51669677db764d35ea4e8097a",
"fuelmain_sha":
"97d7f6d5461db3afc27f58160cf9f6985230d255",
"mirantis":
"yes",
"nailgun_sha":
"4477ba3a6efc4379a6509386e7a9e2e6ae832041",
"ostf_sha":
"a0df6e5e34e6180f2be57034ec90b506f4eb3905",
"production":
"docker",
"release":
"5.0"
}

2014-05-13 13:06:35 ERR
 (/Stage[main]/Keystone/Exec[keystone-manage db_sync]) /usr/lib/ruby/site_ruby/1.8/puppet/transaction/event_manager.rb:101:in `process_callback'
2014-05-13 13:06:35 ERR
 (/Stage[main]/Keystone/Exec[keystone-manage db_sync]) /usr/lib/ruby/site_ruby/1.8/puppet/transaction/event_manager.rb:101:in `send'
2014-05-13 13:06:35 ERR
 (/Stage[main]/Keystone/Exec[keystone-manage db_sync]) /usr/lib/ruby/site_ruby/1.8/puppet/type/exec.rb:537:in `refresh'
2014-05-13 13:06:35 ERR
 (/Stage[main]/Keystone/Exec[keystone-manage db_sync]) /usr/lib/ruby/site_ruby/1.8/puppet/type/exec.rb:120:in `sync'
2014-05-13 13:06:35 ERR
 (/Stage[main]/Keystone/Exec[keystone-manage db_sync]) /usr/lib/ruby/site_ruby/1.8/puppet/util/errors.rb:97:in `fail'
2014-05-13 13:06:35 ERR
 (/Stage[main]/Keystone/Exec[keystone-manage db_sync]) keystone-manage db_sync returned 1 instead of one of [0]
2014-05-13 13:06:35 ERR
 (/Stage[main]/Keystone/Exec[keystone-manage db_sync]) Failed to call refresh: keystone-manage db_sync returned 1 instead of one of [0]

[Bogdan Dobrelya (bogdando)]

I cannot reproduce it on #198

[Bogdan Dobrelya (bogdando)]

RCA from the logs is required as well

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/93601

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/93601
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=3161e6ce1816d9a2e22527fe01a374c3f5c865f1
Submitter: Jenkins
Branch: master

commit 3161e6ce1816d9a2e22527fe01a374c3f5c865f1
Author: Sergey Vasilenko <email address hidden>
Date: Wed May 14 19:33:39 2014 +0400

    Add retries for keystone-manage db_sync and make relationships more strong.

    This retries need for prevent broke deploy, if puppet make decision to restart
    haproxy near this exec.

    Closes-bug: #1319087
    Change-Id: I0e61d2bdf04a11c936ca0bc7b932478fa28f360e

[Andrey Sledzinskiy (asledzinskiy)]

verified on {"build_id": "2014-05-19_01-10-31", "mirantis": "yes", "build_number": "212", "ostf_sha": "353f918197ec53a00127fd28b9151f248a2a2d30", "nailgun_sha": "ab7f7dfddadfe0e08a39693c6d33aa0250f20142", "production": "docker", "api": "1.0", "fuelmain_sha": "9de65bfdb7e8bc7c0ec6d47dfabf4a65f8a9335b", "astute_sha": "a3432e6e31ffd6f1c56386b2eb54afeacb74750b", "release": "5.0", "fuellib_sha": "b4671dcaa93d45ac219991ed3f89b512342c4777"}

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/4.1
Review: https://review.openstack.org/96883

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/96883
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=75d5483551cba4ac20b6ad9153519f418e3ec528
Submitter: Jenkins
Branch: stable/4.1

commit 75d5483551cba4ac20b6ad9153519f418e3ec528
Author: Sergey Vasilenko <email address hidden>
Date: Wed May 14 19:33:39 2014 +0400

    Add retries for keystone-manage db_sync and make relationships more strong.

    This retries need for prevent broke deploy, if puppet make decision to restart
    haproxy near this exec.

    Closes-bug: #1319087
    Change-Id: I0e61d2bdf04a11c936ca0bc7b932478fa28f360e

[Andrey Sledzinskiy (asledzinskiy)]

verified on {

    "build_id": "2014-06-04_09-16-08",
    "mirantis": "yes",
    "build_number": "341",
    "nailgun_sha": "a828d6b7610f872980d5a2113774f1cda6f6810b",
    "ostf_sha": "c959aa55f83fe2555cf2d382559271c7a9b17467",
    "fuelmain_sha": "7ed0f85acc0bab4b9157703a618b8cc9fd7de3e1",
    "astute_sha": "55df06b2e84fa5d71a1cc0e78dbccab5db29d968",
    "release": "4.1B",
    "fuellib_sha": "0e96fc5a340cd57f75c454ea8536471379299494"

}