Race condition to bootstrap Grafana DB

[Doug Szumski]

On a multi-instance Grafana deployment all nodes attempt to bootstrap the DB, one of them wins and the others either don't notice because it's already been done, or fail trying to bootstrap the DB after it's already been done. Observed in CI with Monasca Grafana (based on 4.2) but also reported on vanilla Grafana (6). Low impact to users since the containers should just restart and carry on as normal once the DB is bootstrapped.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/742688

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/742688
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=2c730590d72e7b6d3056c53804f924d43dab8363
Submitter: Zuul
Branch: master

commit 2c730590d72e7b6d3056c53804f924d43dab8363
Author: Doug Szumski <email address hidden>
Date: Thu Jul 23 10:26:27 2020 +0000

    Improve Grafana DB bootstrap

    This fixes an issue where multiple Grafana instances would race
    to bootstrap the Grafana DB. The following changes are made:

    - Only start additional Grafana instances after the DB has been
      configured.

    - During upgrade, don't allow old instances to run with an
      upgraded DB schema.

    Change-Id: I3e0e077ba6a6f43667df042eb593107418a06c39
    Closes-Bug: #1888681

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/743299

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/743300

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/743299
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=4d9650ab2d79540123525486ef17f5185bed120d
Submitter: Zuul
Branch: stable/ussuri

commit 4d9650ab2d79540123525486ef17f5185bed120d
Author: Doug Szumski <email address hidden>
Date: Thu Jul 23 10:26:27 2020 +0000

    Improve Grafana DB bootstrap

    This fixes an issue where multiple Grafana instances would race
    to bootstrap the Grafana DB. The following changes are made:

    - Only start additional Grafana instances after the DB has been
      configured.

    - During upgrade, don't allow old instances to run with an
      upgraded DB schema.

    Change-Id: I3e0e077ba6a6f43667df042eb593107418a06c39
    Closes-Bug: #1888681
    (cherry picked from commit 2c730590d72e7b6d3056c53804f924d43dab8363)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/743300
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=f2031fc8a1c6b985d931532ee532af7fdc3fa266
Submitter: Zuul
Branch: stable/train

commit f2031fc8a1c6b985d931532ee532af7fdc3fa266
Author: Doug Szumski <email address hidden>
Date: Thu Jul 23 10:26:27 2020 +0000

    Improve Grafana DB bootstrap

    This fixes an issue where multiple Grafana instances would race
    to bootstrap the Grafana DB. The following changes are made:

    - Only start additional Grafana instances after the DB has been
      configured.

    - During upgrade, don't allow old instances to run with an
      upgraded DB schema.

    Change-Id: I3e0e077ba6a6f43667df042eb593107418a06c39
    Closes-Bug: #1888681
    (cherry picked from commit 2c730590d72e7b6d3056c53804f924d43dab8363)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 10.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 9.3.0 release.