environment for metrics doesn't start

[ruediger stock]

Brief Description
-----------------
Start gnocchi, ceilometer, oadh and panko fails during application of stx-openstack.

Severity
--------
Major

Steps to Reproduce
------------------
source /etc/platform/openrc
system helm-chart-attribute-modify stx-openstack gnocchi openstack --enabled true
system helm-chart-attribute-modify stx-openstack ceilometer openstack --enabled true
system helm-chart-attribute-modify stx-openstack aodh openstack --enabled true
system helm-chart-attribute-modify stx-openstack panko openstack --enabled true
system application-apply stx-openstack

Expected Behavior
------------------
Services are up and running. E.g. this command works:
OS_CLOUD=openstack_helm openstack metric server version

Actual Behavior
----------------
Applying fails with
+---------------+------------------------------------------+
| Property | Value |
+---------------+------------------------------------------+
| active | True |
| app_version | 1.0-44-centos-stable-versioned |
| created_at | 2020-07-08T10:17:17.823449+00:00 |
| manifest_file | stx-openstack.yaml |
| manifest_name | armada-manifest |
| name | stx-openstack |
| progress | operation aborted, check logs for detail |
| status | apply-failed |
| updated_at | 2020-07-08T12:23:01.363381+00:00 |
+---------------+------------------------------------------+

Reproducibility
---------------
Reproducible 100% on a virtual duplex standard environment

System Configuration
--------------------
Virtual duplex standard

Branch/Pull Time/Commit
-----------------------
Load from /mirror/starlingx/master/centos/flock/20200708T013409Z

Last Pass
---------
Last known good load was 20200611T021306Z

Timestamp/Logs
--------------
https://files.starlingx.kube.cengn.ca/launchpad/1886819

2020-07-08 12:22:59.521 397 ERROR armada.handlers.tiller [-] [chart=openstack-gnocchi]: Error while installing release osh-openstack-gnocchi: grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "release osh-openstack-gnocchi failed: timed out waiting for the condition"
        debug_error_string = "{"created":"@1594210979.520717885","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"release osh-openstack-gnocchi failed: timed out waiting for the condition","grpc_status":2}"
>

Test Activity
-------------
Regression Testing

Workaround
----------
No workaround known

[Ghada Khalil (gkhalil)]

Assigning to the distro.openstack team

[zhipeng liu (zhipengs)]

Root cause has been found.
It seems latest gnocchi need connecting to ceph.
From log, it could not find rados.
aodh, panko are both OK.
ceilometer depends on gnocchi.

We can add "ceph" to profiles in stx-gnocchi.stable_docker_image.
Further check and verification is ongoing.

Zhipeng

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/740390

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/740391

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/740390
Committed: https://git.openstack.org/cgit/starlingx/upstream/commit/?id=e96d8b71778413710cd369cc32c1f2a9ee95e986
Submitter: Zuul
Branch: master

commit e96d8b71778413710cd369cc32c1f2a9ee95e986
Author: Zhipeng Liu <email address hidden>
Date: Fri Jul 10 18:37:25 2020 +0800

    Fix gnocchi-api could not start up issue

    After using python3 to build image, need change related
    profile and customization like we did for other service
    in directives file.

    Test pass for openstack apply with 4 additional services enabled

    Partial-Bug: 1886819

    Change-Id: Ifdfce220b43166345888b1384f62c4ab8f46ba27
    Signed-off-by: Zhipeng Liu <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/740391
Committed: https://git.openstack.org/cgit/starlingx/openstack-armada-app/commit/?id=a8e6da1fe04c66a65b52a9a97366c2f8e5e1cc78
Submitter: Zuul
Branch: master

commit a8e6da1fe04c66a65b52a9a97366c2f8e5e1cc78
Author: Zhipeng Liu <email address hidden>
Date: Fri Jul 10 18:44:09 2020 +0800

    Fix gnocchi-api could not start up issue

    After using python3 to build image, need to change related
    wsgi profile like we did for other openstack service.

    Test pass for openstack apply with 4 additonal services enabled

    Closes-Bug: 1886819

    Depends-on: https://review.opendev.org/#/c/740390/
    Change-Id: Ic7345760a0a387c87f79b78e7949df920da83044
    Signed-off-by: Zhipeng Liu <email address hidden>

[Ghada Khalil (gkhalil)]

@Zhipeng, please cherrypick this to r/stx.4.0

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: r/stx.4.0
Review: https://review.opendev.org/741826

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: r/stx.4.0
Review: https://review.opendev.org/741827

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/741826
Committed: https://git.openstack.org/cgit/starlingx/upstream/commit/?id=bfc2f138634e4487db037d096194c4c086fe541d
Submitter: Zuul
Branch: r/stx.4.0

commit bfc2f138634e4487db037d096194c4c086fe541d
Author: Zhipeng Liu <email address hidden>
Date: Fri Jul 10 18:37:25 2020 +0800

    Fix gnocchi-api could not start up issue

    After using python3 to build image, need change related
    profile and customization like we did for other service
    in directives file.

    Test pass for openstack apply with 4 additional services enabled

    Partial-Bug: 1886819

    Change-Id: Ifdfce220b43166345888b1384f62c4ab8f46ba27
    Signed-off-by: Zhipeng Liu <email address hidden>
    (cherry picked from commit e96d8b71778413710cd369cc32c1f2a9ee95e986)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/741827
Committed: https://git.openstack.org/cgit/starlingx/openstack-armada-app/commit/?id=add8e9a1e81e12e51fc20ac7c68272a5dcf67add
Submitter: Zuul
Branch: r/stx.4.0

commit add8e9a1e81e12e51fc20ac7c68272a5dcf67add
Author: Zhipeng Liu <email address hidden>
Date: Fri Jul 10 18:44:09 2020 +0800

    Fix gnocchi-api could not start up issue

    After using python3 to build image, need to change related
    wsgi profile like we did for other openstack service.

    Test pass for openstack apply with 4 additonal services enabled

    Closes-Bug: 1886819

    Depends-on: https://review.opendev.org/#/c/741826
    Change-Id: Ic7345760a0a387c87f79b78e7949df920da83044
    Signed-off-by: Zhipeng Liu <email address hidden>
    (cherry picked from commit a8e6da1fe04c66a65b52a9a97366c2f8e5e1cc78)

[ruediger stock (rstock)]

The issue can still be reproduced with http://mirror.starlingx.cengn.ca/mirror/starlingx/rc/4.0/centos/flock/20200729T143257Z

The CHANGELOG.TXT states
./cgcs-root/stx/openstack-armada-app add8e9a1e81e12e51fc20ac7c68272a5dcf67add 2020-07-21 01:27:36 +0000 zhipeng liu <email address hidden> Fix gnocchi-api could not start up issue

[ruediger stock (rstock)]

Proven to work fine on master.
controller-0:~$ cat /etc/build.info
###
### StarlingX
### Built from master
###

OS="centos"
SW_VERSION="20.06"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20200730T013417Z"

JOB="STX_build_layer_flock_master_master"
<email address hidden>"
BUILD_NUMBER="197"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2020-07-30 01:34:17 +0000"

FLOCK_OS="centos"
FLOCK_JOB="STX_build_layer_flock_master_master"
<email address hidden>"
FLOCK_BUILD_NUMBER="197"
FLOCK_BUILD_HOST="starlingx_mirror"
FLOCK_BUILD_DATE="2020-07-30 01:34:17 +0000"

[zhipeng liu (zhipengs)]

Hi ruediger,

Another patch seems not yet merged in this daily build
https://review.opendev.org/#/c/741826/

Furthermore，this patch is for gnocchi image build, we usually rebuild service image every Monday.
So, please retest it after next Tuesday. Or, you can ask Scott to manually trigger service image build before
you start test.

Thanks!
Zhipeng

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/upstream/+/792215

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/upstream/+/792215
Committed: https://opendev.org/starlingx/upstream/commit/ab2f84da41b6865e6db05ce0e24bc1d4e0e379ae
Submitter: "Zuul (22348)"
Branch: f/centos8

commit a4046414b634e027f646be58502f3af2ea2329f0
Author: Andy Ning <email address hidden>
Date: Mon Apr 26 16:22:26 2021 -0400

    Enforce "cannot reuse the last 2 passwords" for ks users

    Currently the "unique_last_password_count" attribute in keystone
    configuration is set to "2", which enforces "cannot reuse the last
    1 passwords" in history instead of "cannot reuse the last 2 passwords"
    stated in security document.

    This update changed "unique_last_password_count" attribute to "3" so
    that keystone users password change rule complies with the document.

    Closes-Bug: 1924772
    Change-Id: I6a2de54336c7253022d49ecb118a315a7825c889
    Signed-off-by: Andy Ning <email address hidden>

commit 341eb6980c3a290f3633616bef0f32152a51b41f
Author: Daniel Pereira <email address hidden>
Date: Wed Feb 10 15:44:28 2021 -0300

    Update cinder directives build file

    Currently, cinder docker image doesn't contain nfs mount helpers
    installed, so trying to mount a NFS volume on a cinder-backup
    container fails.
    In order to enable support for NFS backend on cinder-backup, we
    need to install the nfs-utils package on cinder image, so that
    cinder-backup is able to mount NFS volumes.

    Task: 41796
    Story: 2008613
    Change-Id: Ib8e4675069292dc43f98ff55c25626a19ed37b12
    Signed-off-by: Daniel Pereira <email address hidden>

commit d7573c28f9257280239b37985f142cfd416e443c
Author: Chen, Haochuan Z <email address hidden>
Date: Thu Jan 7 13:33:22 2021 +0800

    WA to fix mysqlclient version conflict with Django

    https://stackoverflow.com/questions/55657752/django-installing-mysqlclient-error-mysqlclient-1-3-13-or-newer-is-required
    Fix with guide from stackoverflow. After openstack image
    upgrade to ussuri, Django upgrade to 2.2, which request
    mysqlclient newer than 1.3.13, conflict with version
    0.9.3 in current image. Fix with WA in above link, and
    currently we use pymysql not mysql.

    Closes-Bug: 1908117

    Change-Id: Ic7054c6736993394d92bb0aec25397fd22f84d31
    Signed-off-by: Chen, Haochuan Z <email address hidden>

commit 4a545ec5844cc24a942b5eafd90dfa69ff68a921
Author: Don Penney <email address hidden>
Date: Thu Dec 17 13:21:18 2020 -0500

    Add auto-version for remaining stx/upstream packages

    Update remaining StarlingX packages with hardcoded TIS_PATCH_VER to
    use PKG_GITREVCOUNT where possible, with offsets as needed to ensure
    the version is incremented above the hardcoded version.

    Story: 2008455
    Task: 41458
    Signed-off-by: Don Penney <email address hidden>
    Change-Id: Iaf71fdb3f9c79573ef64f6c82b1a2120d224d959

commit e96d8b71778413710cd369cc32c1f2a9ee95e986
Author: Zhipeng Liu <email address hidden>
Date: Fri Jul 10 18:37:25 2020 +0800

    Fix gnocchi-api could not start up issue

    After using python3 to build image, need change related
 ...