ceph-mgr restful plugin error preventing platform-integ-apps from auto applying

[Chris Winnicki]

Brief Description
-----------------
ceph-mgr restful plugin error preventing platform-integ-apps from auto applying

Severity
--------
Major

Steps to Reproduce
------------------
Install 1+1 AIO duplex-direct
unlock both controllers

Monitor system application-list for platform-integ-apps to be uploaded and applied

Expected Behavior
------------------
platform-integ-apps should be automatically uploaded and applied

Actual Behavior
----------------
platform-integ-apps is stuck in uploaded state due to:
ceph-mgr restful plugin error preventing platform-integ-apps from auto applying

The following exception is seen in the /var/log/sysinv.log

2019-07-09 15:25:28.593 108213 INFO sysinv.common.ceph [req-13799e6b-9b98-41b4-9713-0d4be9fcc2e2 None None] ceph osd crush rule ls
2019-07-09 15:25:29.184 108213 ERROR sysinv.openstack.common.rpc.amqp [req-13799e6b-9b98-41b4-9713-0d4be9fcc2e2 None None] Exception during message handling
2019-07-09 15:25:29.184 108213 TRACE sysinv.openstack.common.rpc.amqp Traceback (most recent call last):
2019-07-09 15:25:29.184 108213 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 438, in _process_data
2019-07-09 15:25:29.184 108213 TRACE sysinv.openstack.common.rpc.amqp **args)
2019-07-09 15:25:29.184 108213 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2019-07-09 15:25:29.184 108213 TRACE sysinv.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2019-07-09 15:25:29.184 108213 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4401, in iplatform_update_by_ihost
2019-07-09 15:25:29.184 108213 TRACE sysinv.openstack.common.rpc.amqp cceph.fix_crushmap(self.dbapi)
2019-07-09 15:25:29.184 108213 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/common/ceph.py", line 718, in fix_crushmap
2019-07-09 15:25:29.184 108213 TRACE sysinv.openstack.common.rpc.amqp rule_is_present, _, _ = _operator._crush_rule_status(default_ceph_tier_name)
2019-07-09 15:25:29.184 108213 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/common/ceph.py", line 55, in _crush_rule_status
2019-07-09 15:25:29.184 108213 TRACE sysinv.openstack.common.rpc.amqp response, body = self._ceph_api.osd_crush_rule_ls(body='json')
2019-07-09 15:25:29.184 108213 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/cephclient/client.py", line 2752, in osd_crush_rule_ls
2019-07-09 15:25:29.184 108213 TRACE sysinv.openstack.common.rpc.amqp body=body, timeout=timeout)
2019-07-09 15:25:29.184 108213 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/cephclient/client.py", line 162, in _request
2019-07-09 15:25:29.184 108213 TRACE sysinv.openstack.common.rpc.amqp self._get_password()
2019-07-09 15:25:29.184 108213 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/cephclient/client.py", line 94, in _get_password
2019-07-09 15:25:29.184 108213 TRACE sysinv.openstack.common.rpc.amqp raise CephMonRestfulMissingUserCredentials(self.username)
2019-07-09 15:25:29.184 108213 TRACE sysinv.openstack.common.rpc.amqp CephMonRestfulMissingUserCredentials: Failed to get ceph-mgr restful plugin credentials for user: admin
2019-07-09 15:25:29.184 108213 TRACE sysinv.openstack.common.rpc.amqp
2019-07-09 15:25:29.187 108213 ERROR sysinv.openstack.common.rpc.common [req-13799e6b-9b98-41b4-9713-0d4be9fcc2e2 None None] Returning exception Failed to get ceph-mgr restful plugin credentials for user: admin to caller

Reproducibility
---------------
not known

System Configuration
--------------------
1+1 AIO duplex-direct

Lab: yow-cgcs-r430-1_2

Branch/Pull Time/Commit
-----------------------
### StarlingX
### Built from master
###

OS="centos"
SW_VERSION="19.01"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20190707T233000Z"

JOB="STX_build_master_master"
<email address hidden>"
BUILD_NUMBER="173"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2019-07-07 23:30:00 +0000"

Last Pass
---------
2019-05-30_16-58-16

Timestamp/Logs
--------------
Attached

Test Activity
-------------
Regression

[Ghada Khalil (gkhalil)]

Assigned to Daniel Badea as requested by Bob Church.
I will let Cindy determine whether this is gating for stx.2.0 with the storage domain owners/TLs.

[Matt Peters (mpeters-wrs)]

The ceph utilities should not be using pecan since they don't know the context in which they are running. There are several places that have the dbapi instance passed as an argument but then using the pecan.request.dbapi incorrectly.

Specifically reported by this LP is get_monitors_status.
This specific issue was introduced by commit b94a088e1fc289f1718d2cd0e78a95cd42909ea2.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/671138

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/671138
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=4fb84ab33a704ed2f932277dd97bad5bcac23eaa
Submitter: Zuul
Branch: master

commit 4fb84ab33a704ed2f932277dd97bad5bcac23eaa
Author: Daniel Badea <email address hidden>
Date: Tue Jul 16 19:06:50 2019 +0000

    Revert "Add checks to deny adding secondary tiers before ceph is up"

    This reverts commit b94a088e1fc289f1718d2cd0e78a95cd42909ea2
    that is fixing bug 1830191 but is causing bug 1835938

    Closes-Bug: 1835938
    Change-Id: If28387318e7bedd7f16a2aebfba25a4fb0989bb8
    Signed-off-by: Daniel Badea <email address hidden>