if db sync fails, it is never retried leading to errors

[Alex Schultz]

In many of our modules we have a db sync tasks that is a refreshonly exec. This exec is only triggered on config changes or package installs. If the db-sync fails, it's never rerun which can lead to weird service errors. We should at least attempt to retry the db-sync process if it fails. This issue is primarily experienced by downstream consumers of the modules who have database connectivity issues mid sync. For example see Bug 1620268

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/379148

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/379166

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/379169

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/379170

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/379171

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/379172

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/379176

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/379183

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/379196

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/379202

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/379206

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/379209

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/379210

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/379211

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/379212

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/379214

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/379217

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/379223

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/379225

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/379226

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/379227

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/379228

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/379231

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/379232

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/379291

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/379297

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/379291
Committed: https://git.openstack.org/cgit/openstack/puppet-congress/commit/?id=e531fa487db83ea9adebec7130d0df56ac2d738d
Submitter: Jenkins
Branch: master

commit e531fa487db83ea9adebec7130d0df56ac2d738d
Author: Sharat Sharma <email address hidden>
Date: Thu Sep 29 14:31:15 2016 +0530

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: I254ed65b1767c55d0bfad6cfc9692c1ac6dfdca8
    Closes-Bug: #1628580

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/379226
Committed: https://git.openstack.org/cgit/openstack/puppet-rally/commit/?id=42033fef85346da279682d26e38bc191f6f1198f
Submitter: Jenkins
Branch: master

commit 42033fef85346da279682d26e38bc191f6f1198f
Author: Sharat Sharma <email address hidden>
Date: Thu Sep 29 12:52:15 2016 +0530

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: Ife3a9bb2851fd730dbd693bda18733ffdec1286e
    Closes-Bug: #1628580

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/379297
Committed: https://git.openstack.org/cgit/openstack/puppet-glare/commit/?id=ad711e929d37d292f0085c3595c2cfcf4a467aca
Submitter: Jenkins
Branch: master

commit ad711e929d37d292f0085c3595c2cfcf4a467aca
Author: Sharat Sharma <email address hidden>
Date: Thu Sep 29 14:38:56 2016 +0530

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: I747ad31ea7095c27f0760aec8f1ccea2195223cc
    Closes-Bug: #1628580

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/379211
Committed: https://git.openstack.org/cgit/openstack/puppet-manila/commit/?id=71619fbfc17540a641ca8f1d793607bfbc1350ab
Submitter: Jenkins
Branch: master

commit 71619fbfc17540a641ca8f1d793607bfbc1350ab
Author: Sharat Sharma <email address hidden>
Date: Thu Sep 29 12:36:52 2016 +0530

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: I64e011820591796168cafa4e0a93d5e2157ce445
    Closes-Bug: #1628580

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/379212
Committed: https://git.openstack.org/cgit/openstack/puppet-murano/commit/?id=e7db0e93f3edeb6d5857fbd5149a89a9b10f63a2
Submitter: Jenkins
Branch: master

commit e7db0e93f3edeb6d5857fbd5149a89a9b10f63a2
Author: Sharat Sharma <email address hidden>
Date: Thu Sep 29 12:38:35 2016 +0530

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: I1ee9f046284d8720447c02f0fbbbc4448eaaf4d9
    Closes-Bug: #1628580

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/379231
Committed: https://git.openstack.org/cgit/openstack/puppet-vitrage/commit/?id=2c0dcf88a1a0fd7e29df5452b0590842e960f388
Submitter: Jenkins
Branch: master

commit 2c0dcf88a1a0fd7e29df5452b0590842e960f388
Author: Sharat Sharma <email address hidden>
Date: Thu Sep 29 12:59:03 2016 +0530

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: Ic6094bb265879bd6ad14beeb3a966a79d4d3b3ea
    Closes-Bug: #1628580

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/379210
Committed: https://git.openstack.org/cgit/openstack/puppet-magnum/commit/?id=cab5e191ad027a9f7be9d6c09f15e0ad0d8232bd
Submitter: Jenkins
Branch: master

commit cab5e191ad027a9f7be9d6c09f15e0ad0d8232bd
Author: Sharat Sharma <email address hidden>
Date: Thu Sep 29 12:33:04 2016 +0530

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: I983008e9cdd46f64ee91712c602712d4eed31714
    Closes-Bug: #1628580

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/379225
Committed: https://git.openstack.org/cgit/openstack/puppet-panko/commit/?id=1cf9c3f3c1dc816b79d65dc71f65653d3389b5c9
Submitter: Jenkins
Branch: master

commit 1cf9c3f3c1dc816b79d65dc71f65653d3389b5c9
Author: Sharat Sharma <email address hidden>
Date: Thu Sep 29 12:50:25 2016 +0530

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: Ifb01a82e74d3ab46f76d1f94a685e96c86119258
    Closes-Bug: #1628580

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/379166
Committed: https://git.openstack.org/cgit/openstack/puppet-aodh/commit/?id=bac067fff5a8194a7093e29a1845450a4e329183
Submitter: Jenkins
Branch: master

commit bac067fff5a8194a7093e29a1845450a4e329183
Author: Sharat Sharma <email address hidden>
Date: Thu Sep 29 11:36:34 2016 +0530

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: I6a19e10ec305f54232293fd9807217291bf50875
    Closes-Bug: #1628580

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/379176
Committed: https://git.openstack.org/cgit/openstack/puppet-ec2api/commit/?id=f52c48bf6ebccc80b57e34715cf26806944ee499
Submitter: Jenkins
Branch: master

commit f52c48bf6ebccc80b57e34715cf26806944ee499
Author: Sharat Sharma <email address hidden>
Date: Thu Sep 29 11:57:06 2016 +0530

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: I435ff4ce72e36a53f06a6300d66baf202175ca95
    Closes-Bug: #1628580

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/379196
Committed: https://git.openstack.org/cgit/openstack/puppet-gnocchi/commit/?id=81c45cdc0857a195cd2f6a760f269ad0cb1c04bd
Submitter: Jenkins
Branch: master

commit 81c45cdc0857a195cd2f6a760f269ad0cb1c04bd
Author: Sharat Sharma <email address hidden>
Date: Thu Sep 29 12:21:44 2016 +0530

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: Ia6a095c1bc0633a9369a7cf3f61d994d156ddd78
    Closes-Bug: #1628580

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/379223
Committed: https://git.openstack.org/cgit/openstack/puppet-octavia/commit/?id=2b383a087f8cbdfb49093799b6d5ee54de274799
Submitter: Jenkins
Branch: master

commit 2b383a087f8cbdfb49093799b6d5ee54de274799
Author: Sharat Sharma <email address hidden>
Date: Thu Sep 29 12:47:36 2016 +0530

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: I892869a67136b648a0acf9d4468914c2d8875f86
    Closes-Bug: #1628580

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/379171
Committed: https://git.openstack.org/cgit/openstack/puppet-cinder/commit/?id=a36d4f91a1f737a2baa4719e8f356fc43793f37f
Submitter: Jenkins
Branch: master

commit a36d4f91a1f737a2baa4719e8f356fc43793f37f
Author: Sharat Sharma <email address hidden>
Date: Thu Sep 29 11:47:17 2016 +0530

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: I108a33ed8bacc12c9b7cc00e603c78ea51c948b2
    Closes-Bug: #1628580

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/379170
Committed: https://git.openstack.org/cgit/openstack/puppet-ceilometer/commit/?id=b7d3c18b2a094be683138a3743480f83f48c5add
Submitter: Jenkins
Branch: master

commit b7d3c18b2a094be683138a3743480f83f48c5add
Author: Sharat Sharma <email address hidden>
Date: Thu Sep 29 11:43:11 2016 +0530

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: I6117fc818c44ac31ae656c18ed7fe5434fa73b77
    Closes-Bug: #1628580

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/407093

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/406864
Committed: https://git.openstack.org/cgit/openstack/puppet-nova/commit/?id=9bc89b056e16c53fc13fce12224a07743880d2dc
Submitter: Jenkins
Branch: master

commit 9bc89b056e16c53fc13fce12224a07743880d2dc
Author: Anton Chevychalov <email address hidden>
Date: Mon Dec 5 13:02:51 2016 +0300

    Added retries in db_sync for nova_api db

    That patch continue fixes made for LP#1628580
    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: Ie718efef81a24d0299fff83d39d04fac5a36fbb0
    Related-Bug: #1628580
    Closes-Bug: #1647305

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/407088
Committed: https://git.openstack.org/cgit/openstack/puppet-murano/commit/?id=99fadd681e083a02a12bd0e08f9ad811c95466f3
Submitter: Jenkins
Branch: stable/newton

commit 99fadd681e083a02a12bd0e08f9ad811c95466f3
Author: Sharat Sharma <email address hidden>
Date: Thu Sep 29 12:38:35 2016 +0530

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: I1ee9f046284d8720447c02f0fbbbc4448eaaf4d9
    Related-Bug: #1628580
    Closes-Bug: #1645474
    (cherry picked from commit e7db0e93f3edeb6d5857fbd5149a89a9b10f63a2)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/407082
Committed: https://git.openstack.org/cgit/openstack/puppet-heat/commit/?id=5cfe50fda46646ade08b2e0c2cb536cbf59dcb21
Submitter: Jenkins
Branch: stable/newton

commit 5cfe50fda46646ade08b2e0c2cb536cbf59dcb21
Author: Sharat Sharma <email address hidden>
Date: Thu Sep 29 12:23:49 2016 +0530

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: I094f592989928cefef0b9cd9c7b9d837f51d03b1
    Related-Bug: #1628580
    Closes-Bug: #1645474
    (cherry picked from commit 5e3b5e050fa2b9377792852aabba8487abb7099e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/407080
Committed: https://git.openstack.org/cgit/openstack/puppet-glance/commit/?id=138fd2917e54d17ff6cb4f379342aa65b6b5dde7
Submitter: Jenkins
Branch: stable/newton

commit 138fd2917e54d17ff6cb4f379342aa65b6b5dde7
Author: Sharat Sharma <email address hidden>
Date: Thu Sep 29 12:11:05 2016 +0530

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: I8778bd1b609bc5458a250111dc0a0f9b2fb52161
    Related-Bug: #1628580
    Closes-Bug: #1645474
    (cherry picked from commit 0fc2099d127b3a5d459f5591f4380d110cc92e4c)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/407339

[OpenStack Infra (hudson-openstack)]

Change abandoned by Anton Chevychalov (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/407027
Reason: Per Alex request

[OpenStack Infra (hudson-openstack)]

Change abandoned by Anton Chevychalov (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/407028
Reason: Per Alex request

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/407339
Committed: https://git.openstack.org/cgit/openstack/puppet-congress/commit/?id=d6ff0c15c43b84ebff8d0ae43a07a41330d4983d
Submitter: Jenkins
Branch: master

commit d6ff0c15c43b84ebff8d0ae43a07a41330d4983d
Author: ZhongShengping <[e-mail]>
Date: Tue Dec 6 14:34:10 2016 +0800

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: Ieb5f39dbc53bb7a1a2031eb139bb3d8cda579cb4
    Closes-Bug: #1628580

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/407087
Committed: https://git.openstack.org/cgit/openstack/puppet-murano/commit/?id=8951da4f8495f5b2e47549414df226f8f15465b8
Submitter: Jenkins
Branch: stable/mitaka

commit 8951da4f8495f5b2e47549414df226f8f15465b8
Author: Sharat Sharma <email address hidden>
Date: Thu Sep 29 12:38:35 2016 +0530

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: I1ee9f046284d8720447c02f0fbbbc4448eaaf4d9
    Related-Bug: #1628580
    Closes-Bug: #1645474
    (cherry picked from commit e7db0e93f3edeb6d5857fbd5149a89a9b10f63a2)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/407073
Committed: https://git.openstack.org/cgit/openstack/puppet-ceilometer/commit/?id=e8aa7064ad7dde657ee2f79bc3aad5e130d46c02
Submitter: Jenkins
Branch: stable/mitaka

commit e8aa7064ad7dde657ee2f79bc3aad5e130d46c02
Author: Sharat Sharma <email address hidden>
Date: Thu Sep 29 11:43:11 2016 +0530

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: I6117fc818c44ac31ae656c18ed7fe5434fa73b77
    Related-Bug: #1628580
    Closes-Bug: #1645474
    (cherry picked from commit b7d3c18b2a094be683138a3743480f83f48c5add)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/407091
Committed: https://git.openstack.org/cgit/openstack/puppet-neutron/commit/?id=e314c48c23330081d34d99c8fc8ab016dac1c642
Submitter: Jenkins
Branch: stable/newton

commit e314c48c23330081d34d99c8fc8ab016dac1c642
Author: Sharat Sharma <email address hidden>
Date: Thu Sep 29 12:40:33 2016 +0530

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: I836b56d125ec781f24b5ea1797ac882b353e2379
    Related-Bug: #1628580
    Closes-Bug: #1645474
    (cherry picked from commit 9e7ed558f22572f4a99534ff57ab1ccc3eedc787)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/407078
Committed: https://git.openstack.org/cgit/openstack/puppet-glance/commit/?id=1dd7369c2ba3f5bfbda0ceb2bdc9ad73f5e25203
Submitter: Jenkins
Branch: stable/mitaka

commit 1dd7369c2ba3f5bfbda0ceb2bdc9ad73f5e25203
Author: Sharat Sharma <email address hidden>
Date: Thu Sep 29 12:11:05 2016 +0530

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: I8778bd1b609bc5458a250111dc0a0f9b2fb52161
    Related-Bug: #1628580
    Closes-Bug: #1645474
    (cherry picked from commit 0fc2099d127b3a5d459f5591f4380d110cc92e4c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/407086
Committed: https://git.openstack.org/cgit/openstack/puppet-keystone/commit/?id=678aace5615c142eda77f86ceb1ef80d941229e9
Submitter: Jenkins
Branch: stable/newton

commit 678aace5615c142eda77f86ceb1ef80d941229e9
Author: Sharat Sharma <email address hidden>
Date: Thu Sep 29 12:31:22 2016 +0530

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: Ia9d09515435618624919ad715ffa3df833a50a7a
    Related-Bug: #1628580
    Closes-Bug: #1645474
    (cherry picked from commit 28c04297b2751f3d5dea152cadc1469033d7e1c7)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/407081
Committed: https://git.openstack.org/cgit/openstack/puppet-heat/commit/?id=b5182c958845b5579583b4278d11713b48d35015
Submitter: Jenkins
Branch: stable/mitaka

commit b5182c958845b5579583b4278d11713b48d35015
Author: Sharat Sharma <email address hidden>
Date: Thu Sep 29 12:23:49 2016 +0530

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: I094f592989928cefef0b9cd9c7b9d837f51d03b1
    Related-Bug: #1628580
    Closes-Bug: #1645474
    (cherry picked from commit 5e3b5e050fa2b9377792852aabba8487abb7099e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/407090
Committed: https://git.openstack.org/cgit/openstack/puppet-neutron/commit/?id=6857bc69adaa21ce173a6809563ba1515b2add23
Submitter: Jenkins
Branch: stable/mitaka

commit 6857bc69adaa21ce173a6809563ba1515b2add23
Author: Sharat Sharma <email address hidden>
Date: Thu Sep 29 12:40:33 2016 +0530

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: I836b56d125ec781f24b5ea1797ac882b353e2379
    Related-Bug: #1628580
    Closes-Bug: #1645474
    (cherry picked from commit 9e7ed558f22572f4a99534ff57ab1ccc3eedc787)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/407076
Committed: https://git.openstack.org/cgit/openstack/puppet-cinder/commit/?id=f4ff62fe9b143b24caa5f93331ca0a738b75f78d
Submitter: Jenkins
Branch: stable/mitaka

commit f4ff62fe9b143b24caa5f93331ca0a738b75f78d
Author: Sharat Sharma <email address hidden>
Date: Thu Sep 29 11:47:17 2016 +0530

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: I108a33ed8bacc12c9b7cc00e603c78ea51c948b2
    Related-Bug: #1628580
    Closes-Bug: #1645474
    (cherry picked from commit a36d4f91a1f737a2baa4719e8f356fc43793f37f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/407084
Committed: https://git.openstack.org/cgit/openstack/puppet-ironic/commit/?id=f8b8cfc9229ee4f8903111fb622627c08408f3e3
Submitter: Jenkins
Branch: stable/newton

commit f8b8cfc9229ee4f8903111fb622627c08408f3e3
Author: Sharat Sharma <email address hidden>
Date: Thu Sep 29 12:28:16 2016 +0530

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: I00051bf680df897fa039fe63a737c518924f448e
    Related-Bug: #1628580
    Closes-Bug: #1645474
    (cherry picked from commit a2cfc7e7778ab2dc8316d613fd014a664940609d)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/407092
Committed: https://git.openstack.org/cgit/openstack/puppet-sahara/commit/?id=4e6375b0472cfbc93ebf688accfa962ff1664270
Submitter: Jenkins
Branch: stable/mitaka

commit 4e6375b0472cfbc93ebf688accfa962ff1664270
Author: Sharat Sharma <email address hidden>
Date: Thu Sep 29 12:53:54 2016 +0530

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: Ib7162c07542d9c8c81ed35f931a2b91281610774
    Related-Bug: #1628580
    Closes-Bug: #1645474
    (cherry picked from commit 3f8e5d1c542dda2bc261c74fa1dad1d3289aa686)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/407070
Committed: https://git.openstack.org/cgit/openstack/puppet-aodh/commit/?id=ad27c5b2d446c75b1d2db5d5d50dc2e6f5c4751f
Submitter: Jenkins
Branch: stable/mitaka

commit ad27c5b2d446c75b1d2db5d5d50dc2e6f5c4751f
Author: Sharat Sharma <email address hidden>
Date: Thu Sep 29 11:36:34 2016 +0530

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: I6a19e10ec305f54232293fd9807217291bf50875
    Related-Bug: #1628580
    Closes-Bug: #1645474
    (cherry picked from commit bac067fff5a8194a7093e29a1845450a4e329183)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/407093
Committed: https://git.openstack.org/cgit/openstack/puppet-sahara/commit/?id=e203d5b6f52611767815b174a7b4a9eff9809ff2
Submitter: Jenkins
Branch: stable/newton

commit e203d5b6f52611767815b174a7b4a9eff9809ff2
Author: Sharat Sharma <email address hidden>
Date: Thu Sep 29 12:53:54 2016 +0530

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: Ib7162c07542d9c8c81ed35f931a2b91281610774
    Related-Bug: #1628580
    Closes-Bug: #1645474
    (cherry picked from commit 3f8e5d1c542dda2bc261c74fa1dad1d3289aa686)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/407028
Committed: https://git.openstack.org/cgit/openstack/puppet-nova/commit/?id=6e71c1c45d04b945df57d7dd0426c7b38799ef82
Submitter: Jenkins
Branch: stable/mitaka

commit 6e71c1c45d04b945df57d7dd0426c7b38799ef82
Author: Anton Chevychalov <email address hidden>
Date: Mon Dec 5 13:02:51 2016 +0300

    Added retries in db_sync for nova_api db

    That patch continue fixes made for LP#1628580
    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: Ie718efef81a24d0299fff83d39d04fac5a36fbb0
    Related-Bug: #1628580
    Related-Bug: #1647305
    Closes-Bug: #1645474

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/407027
Committed: https://git.openstack.org/cgit/openstack/puppet-nova/commit/?id=6cf54de34231157543b216345a7d1e607ea0b286
Submitter: Jenkins
Branch: stable/newton

commit 6cf54de34231157543b216345a7d1e607ea0b286
Author: Anton Chevychalov <email address hidden>
Date: Mon Dec 5 13:02:51 2016 +0300

    Added retries in db_sync for nova_api db

    That patch continue fixes made for LP#1628580
    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: Ie718efef81a24d0299fff83d39d04fac5a36fbb0
    Related-Bug: #1628580
    Related-Bug: #1647305
    Closes-Bug: #1645474

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/409611

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/409611
Committed: https://git.openstack.org/cgit/openstack/puppet-glare/commit/?id=af276afb5f310768f4783873efe93796255e43b0
Submitter: Jenkins
Branch: master

commit af276afb5f310768f4783873efe93796255e43b0
Author: ZhongShengping <[e-mail]>
Date: Mon Dec 12 13:36:18 2016 +0800

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: I3eb965550b53e7cedffc58fddfc8e8cd96b9859f
    Closes-Bug: #1628580

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/407069
Committed: https://git.openstack.org/cgit/openstack/puppet-aodh/commit/?id=45f6e16888edd231de105455c18a4176bf526483
Submitter: Jenkins
Branch: stable/newton

commit 45f6e16888edd231de105455c18a4176bf526483
Author: Sharat Sharma <email address hidden>
Date: Thu Sep 29 11:36:34 2016 +0530

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: I6a19e10ec305f54232293fd9807217291bf50875
    Related-Bug: #1628580
    Closes-Bug: #1645474
    (cherry picked from commit bac067fff5a8194a7093e29a1845450a4e329183)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/407074
Committed: https://git.openstack.org/cgit/openstack/puppet-ceilometer/commit/?id=05a2dc2897b20c4de8fb4a1811527d716f41ede2
Submitter: Jenkins
Branch: stable/newton

commit 05a2dc2897b20c4de8fb4a1811527d716f41ede2
Author: Sharat Sharma <email address hidden>
Date: Thu Sep 29 11:43:11 2016 +0530

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: I6117fc818c44ac31ae656c18ed7fe5434fa73b77
    Related-Bug: #1628580
    Closes-Bug: #1645474
    (cherry picked from commit b7d3c18b2a094be683138a3743480f83f48c5add)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/407077
Committed: https://git.openstack.org/cgit/openstack/puppet-cinder/commit/?id=8460bbec01a8800e5f68501df2462027b98d3924
Submitter: Jenkins
Branch: stable/newton

commit 8460bbec01a8800e5f68501df2462027b98d3924
Author: Sharat Sharma <email address hidden>
Date: Thu Sep 29 11:47:17 2016 +0530

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: I108a33ed8bacc12c9b7cc00e603c78ea51c948b2
    Related-Bug: #1628580
    Closes-Bug: #1645474
    (cherry picked from commit a36d4f91a1f737a2baa4719e8f356fc43793f37f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/407083
Committed: https://git.openstack.org/cgit/openstack/puppet-ironic/commit/?id=b5a2d56ae21f96f5c34942f86774ce259e273984
Submitter: Jenkins
Branch: stable/mitaka

commit b5a2d56ae21f96f5c34942f86774ce259e273984
Author: Sharat Sharma <email address hidden>
Date: Thu Sep 29 12:28:16 2016 +0530

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: I00051bf680df897fa039fe63a737c518924f448e
    Related-Bug: #1628580
    Closes-Bug: #1645474
    (cherry picked from commit a2cfc7e7778ab2dc8316d613fd014a664940609d)

[Mike Bayer (zzzeek)]

I'm also experiencing db syncs that don't fail at all but take longer than the puppet default timeout of 300 seconds (granted this is on an all-in-one VM setup with multiple controllers). Increasing this timeout by adding "timeout => N" to each exec and at least making it configurable would allow this to be fixed. As the migration scripts for components get longer and longer, and projects don't prune their early migration scripts (e.g. like the Nova 216 havana script), this time will get longer and longer and impact more kinds of environments. The fix here that does a "retry" will probably alleviate this to some degree as if you retry a migration script that only got through half its migrations, it will pick up where it left off, but even better would be to allow the script to complete in one run. Would tripleo be open to additional gerrits in this regard?

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/407072
Committed: https://git.openstack.org/cgit/openstack/puppet-barbican/commit/?id=cefec8258d1bb95a461eeb2396deead89864ae66
Submitter: Jenkins
Branch: stable/newton

commit cefec8258d1bb95a461eeb2396deead89864ae66
Author: Sharat Sharma <email address hidden>
Date: Thu Sep 29 11:39:40 2016 +0530

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: I38f0c42faa1dba167bd7b37c63f13ba64f1bb7db
    Related-Bug: #1628580
    Closes-Bug: #1645474
    (cherry picked from commit 93f885f0c13e51f68505fb6b6d4ba158ad9932b4)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/407085
Committed: https://git.openstack.org/cgit/openstack/puppet-keystone/commit/?id=b373de440c1441173a34cd8b60747909d2a55bad
Submitter: Jenkins
Branch: stable/mitaka

commit b373de440c1441173a34cd8b60747909d2a55bad
Author: Sharat Sharma <email address hidden>
Date: Thu Sep 29 12:31:22 2016 +0530

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: Ia9d09515435618624919ad715ffa3df833a50a7a
    Related-Bug: #1628580
    Closes-Bug: #1645474
    (cherry picked from commit 28c04297b2751f3d5dea152cadc1469033d7e1c7)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/553668
Committed: https://git.openstack.org/cgit/openstack/puppet-watcher/commit/?id=c13a788dd99a515a8bc5fc52b653fead39662eb8
Submitter: Zuul
Branch: master

commit c13a788dd99a515a8bc5fc52b653fead39662eb8
Author: ZhongShengping <[e-mail]>
Date: Fri Mar 16 11:21:20 2018 +0800

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: I784ca2fc0a9a906633ad60d1338c4b4b73c376c7
    Closes-Bug: #1628580

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/553839

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/553840

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/553839
Committed: https://git.openstack.org/cgit/openstack/puppet-watcher/commit/?id=194824db7a2845ef12570b8ebb7be1a564fe7dbd
Submitter: Zuul
Branch: stable/queens

commit 194824db7a2845ef12570b8ebb7be1a564fe7dbd
Author: ZhongShengping <[e-mail]>
Date: Fri Mar 16 11:21:20 2018 +0800

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: I784ca2fc0a9a906633ad60d1338c4b4b73c376c7
    Closes-Bug: #1628580
    (cherry picked from commit c13a788dd99a515a8bc5fc52b653fead39662eb8)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/553840
Committed: https://git.openstack.org/cgit/openstack/puppet-watcher/commit/?id=f654140f95ed409e164326d57c7eed681bc3cda2
Submitter: Zuul
Branch: stable/pike

commit f654140f95ed409e164326d57c7eed681bc3cda2
Author: ZhongShengping <[e-mail]>
Date: Fri Mar 16 11:21:20 2018 +0800

    Added retries for db_sync

    If db sync fails, it is never retried leading to errors. So, this
    patch adds retries for db_sync to avoid sync fails.

    Change-Id: I784ca2fc0a9a906633ad60d1338c4b4b73c376c7
    Closes-Bug: #1628580
    (cherry picked from commit c13a788dd99a515a8bc5fc52b653fead39662eb8)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-watcher 12.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-watcher 13.0.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-watcher 11.6.0 release.