gnocchi measures storage is not configured properly on controller-1

[Angie Wang]

Brief Description
-----------------
Gnocchi measures storage is not configured properly on controller-1, missing incoming storages(/opt/gnocchi/incoming*) on controller-1.

Here is the log looping in the /var/log/gnocchi/metricd.log on controller-1 when controller-1 is the active controller:
2018-08-21 07:30:45,590 [22921] INFO gnocchi.service: Gnocchi version 4.2.5
2018-08-21 07:30:45,645 [23133] ERROR gnocchi.cli.metricd: Unexpected error during reporting job
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/gnocchi/cli/metricd.py", line 83, in run
    self._run_job()
  File "/usr/lib/python2.7/site-packages/gnocchi/cli/metricd.py", line 123, in _run_job
    report = self.incoming.measures_report(details=False)
  File "/usr/lib/python2.7/site-packages/gnocchi/incoming/__init__.py", line 145, in measures_report
    metrics, measures, full_details = self._build_report(details)
  File "/usr/lib/python2.7/site-packages/gnocchi/incoming/file.py", line 111, in _build_report
    for i in six.moves.range(self.NUM_SACKS):
  File "/usr/lib/python2.7/site-packages/gnocchi/incoming/__init__.py", line 56, in NUM_SACKS
    raise SackDetectionError(e)
SackDetectionError: [Errno 2] No such file or directory: '/opt/gnocchi/tmp/gnocchi-config'
2018-08-21 07:30:46,484 [23132] ERROR cotyledon._utils: Unhandled exception
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cotyledon/_utils.py", line 84, in exit_on_exception
    yield
  File "/usr/lib/python2.7/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/usr/lib/python2.7/site-packages/gnocchi/cli/metricd.py", line 76, in run
    self._configure()
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 171, in wrapped_f
    return self.call(f, *args, **kw)
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 248, in call
    start_time=start_time)
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 203, in iter
    return fut.result()
  File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 398, in result
    return self.__get_result()
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 251, in call
    result = fn(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/gnocchi/cli/metricd.py", line 161, in _configure
    six.moves.range(self.incoming.NUM_SACKS))
  File "/usr/lib/python2.7/site-packages/gnocchi/incoming/__init__.py", line 56, in NUM_SACKS
    raise SackDetectionError(e)

Steps to Reproduce
------------------
1. Install and configure two controllers
2. Swact to controller-1 after controller-1 is enabled, unlocked

Expected Behavior
------------------
No log flooding and incoming storage should be created in /opt/gnocchi/.

Actual Behavior
----------------
The above log is generated every 1 minute and no incoming storages are created.

Reproducibility
---------------
100% reproducible

System Configuration
--------------------
2 System Controllers

Branch/Pull Time/Commit
-----------------------
master

Timestamp/Logs
--------------
N/A

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/596915

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/596915
Committed: https://git.openstack.org/cgit/openstack/stx-config/commit/?id=adf91e29612e7ab638c65592bdbebed8d4df7692
Submitter: Zuul
Branch: master

commit adf91e29612e7ab638c65592bdbebed8d4df7692
Author: Angie Wang <email address hidden>
Date: Mon Aug 27 16:35:09 2018 -0400

    Configure gnocchi measures storage on both controllers

    Measures cannot be stored and processed when controller-1 is the
    active controller because gnocchi measures storage is not fully
    configured on controller-1. This commit is to run gnocchi-upgrade
    to configure measures storage on both controllers if the storage has
    not been configured.

    Closes-Bug: 1789287

    Change-Id: Iffe7d9f40be049be99755768f7b133d74310635c
    Signed-off-by: Angie Wang <email address hidden>