Ansible: Secondary storage tiers fail to be added before initial unlock

[Ovidiu Poncea]

Brief Description
-----------------

On controller-0, at initial configuration, adding secondary tiers fails. The reason is that ceph has to be up in order for secondary tiers and secondary storage backends to be added. This operation should be denied through a semantic checks. User is permited to add secondary tiers only after Ceph is up.

Displayed error message:

[root@controller-0 wrsroot(keystone_admin)]# system storage-tier-add ceph_cluster gold
Failed to get ceph-mgr restful plugin keys. Command 'ceph restful list-keys --connect-timeout 5' returned non-zero exit status 1

In sysinv.log:

2019-05-15 15:38:33.622 79264 ERROR wsme.api [-] Server-side error: "Failed to get ceph-mgr restful plugin keys. Command 'ceph restful list-keys --connect-timeout 5' returned non-zero exit status 1". Detail:
Traceback (most recent call last):

  File "/usr/lib/python2.7/site-packages/wsmeext/pecan.py", line 85, in callfunction
    result = f(self, *args, **kwargs)

  File "/usr/lib64/python2.7/site-packages/sysinv/api/controllers/v1/storage_tier.py", line 261, in post
    new_tier = _create(self, tier)

  File "/usr/lib64/python2.7/site-packages/sysinv/api/controllers/v1/storage_tier.py", line 505, in _create
    self._ceph.set_crushmap()

  File "/usr/lib64/python2.7/site-packages/sysinv/common/ceph.py", line 551, in set_crushmap
    if fix_crushmap():

  File "/usr/lib64/python2.7/site-packages/sysinv/common/ceph.py", line 731, in fix_crushmap
    rule_is_present, __, __ = _operator._crush_rule_status(default_ceph_tier_name)

  File "/usr/lib64/python2.7/site-packages/sysinv/common/ceph.py", line 55, in _crush_rule_status
    response, body = self._ceph_api.osd_crush_rule_ls(body='json')

  File "/usr/lib/python2.7/site-packages/cephclient/client.py", line 2752, in osd_crush_rule_ls
    body=body, timeout=timeout)

  File "/usr/lib/python2.7/site-packages/cephclient/client.py", line 162, in _request
    self._get_password()

  File "/usr/lib/python2.7/site-packages/cephclient/client.py", line 86, in _get_password
    raise CephMonRestfulListKeysError(str(e))

Severity
--------
Minor: This is not the normal flow.

Steps to Reproduce
------------------
Write down the steps to reproduce the issue

Expected Behavior
------------------
Write down what was expected after taking the steps written above

Actual Behavior
----------------
State what is the actual behavior

Reproducibility
---------------
100% reproducible

System Configuration
--------------------
One node system, Two node system

Branch/Pull Time/Commit
-----------------------
master

Last Pass
---------
before Ansible

Test Activity
-------------
Developer Testing

[Ghada Khalil (gkhalil)]

Marking as release gating; medium priority as the issue is specific to a particular workflow (adding storage tiers). Related to ansible deployment.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/667963

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/667963
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=b94a088e1fc289f1718d2cd0e78a95cd42909ea2
Submitter: Zuul
Branch: master

commit b94a088e1fc289f1718d2cd0e78a95cd42909ea2
Author: Ovidiu Poncea <email address hidden>
Date: Thu Jun 27 17:17:36 2019 +0300

    Add checks to deny adding secondary tiers before ceph is up

    On controller-0, at initial configuration, adding secondary
    tiers fails. The reason is that ceph has to be up in order
    for secondary tiers and secondary storage backends to be
    added. This operation is now denied through a semantic
    checks.

    Another semantic check is added to verify if ceph is
    functional before allowing creation of secondary tiers.

    Change-Id: I75c2e9540c98a01044bdd2666e0eac821f385cfa
    Closes-Bug: 1830191
    Signed-off-by: Ovidiu Poncea <email address hidden>

[Ghada Khalil (gkhalil)]

The above commit is being reverted by:
https://review.opendev.org/#/c/671138/

as it introduced issues in sanity and lab bring-up:
https://bugs.launchpad.net/starlingx/+bug/1835938
https://bugs.launchpad.net/starlingx/+bug/1836758

Re-opening this bug as the fix will need to be re-worked.

[Ghada Khalil (gkhalil)]

See the note from Matt Peters:
https://bugs.launchpad.net/starlingx/+bug/1835938/comments/3

It looks like a bigger change is needed in the ceph utilities beyond the commit above which was reverted.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/671489

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/671489
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=734657dcf8be646bc7594b4a5fba7e2d4c86d571
Submitter: Zuul
Branch: master

commit 734657dcf8be646bc7594b4a5fba7e2d4c86d571
Author: Ovidiu Poncea <email address hidden>
Date: Thu Jul 18 15:21:53 2019 +0300

    Add checks to deny adding secondary tiers before ceph is up

    On controller-0, at initial configuration, adding secondary
    tiers fails. The reason is that ceph has to be up in order
    for secondary tiers and secondary storage backends to be
    added. This operation is now denied through a semantic
    checks.

    Another semantic check is added to verify if ceph is
    functional before allowing creation of secondary tiers.

    Note:
    This is re-adding b94a088e1fc289f1718d2cd0e78a95cd42909ea2
    reverted by: 4fb84ab33a704ed2f932277dd97bad5bcac23eaa

    The difference between the two is the replace of:
    if cutils.is_aio_system(pecan.request.dbapi):
    with
    if cutils.is_aio_system(db_api):
    in get_monitor_status function.

    The problem was that get_monitor_status failed when called
    from sysinv-conductor as it got an invalid context.

    Change-Id: Iaac2e9540c98a01044bdd2666e0eac821f385cfb
    Closes-Bug: 1830191
    Signed-off-by: Ovidiu Poncea <email address hidden>