cinder backup stuck in Error if created on controller-1

[mhg]

Brief Description
-----------------
On controller-1, cinder-backup created using CLI went and remained in "error" status. "cinder backup-show" displayed "An unknown exception occurred."
No file(s) generated under /opt/backups

Severity
--------
Minor

Steps to Reproduce
------------------
1 make sure controller-1 is the active controller
2 execute CLI:
cinder backup-create --container cinder --name <bk_name> <vol_id>

Expected Behavior
------------------
a "cinder-backup" was expected to be created and eventually reached 'available' status; files for the volume were generated under /opt/backups

Actual Behavior
----------------
The "cinder-backup" went and remained in "error". "cinder backup-show" displayed "An unknown exception occurred."
No file(s) generated under /opt/backups
Traceback/errors were found in log files.

Reproducibility
---------------
Reproducible

System Configuration
--------------------
Dedicated storage

Branch/Pull Time/Commit
-----------------------
master as of build-date-time
StarlingX_Upstream as of 2019-01-02_20-18-00

Timestamp/Logs
Jan 3 10:46

--------------

[Ghada Khalil (gkhalil)]

From Wei Zhou:
The root cause is that the empty file /etc/ceph/ceph.client.None.keyring which is a workaround for an upstream bug in rbd is not created on the standby controller because of commit http://git.openstack.org/cgit/openstack/stx-config/commit/?id=2a99d0b4b95348ea849ed6f6f1e7b34f76e2ec84. In the test cinder backup was done after controller swact which means on the newly active controller file /etc/ceph/ceph.client.None.keyring doesn't exist; therefore cinder backup failed.

[Ghada Khalil (gkhalil)]

Marking as release gating; issue was recently introduced by other changes.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/630381
Committed: https://git.openstack.org/cgit/openstack/stx-config/commit/?id=3b4acbfc38f01f9e600d7fb41f0dbd55ba2166e4
Submitter: Zuul
Branch: master

commit 3b4acbfc38f01f9e600d7fb41f0dbd55ba2166e4
Author: Wei Zhou <email address hidden>
Date: Fri Jan 11 15:26:20 2019 -0500

    /etc/ceph/ceph.client.None.keyring not created on standby controller

    This commit ensures that file /etc/ceph/ceph.client.None.keyring is
    created on both controllers after ceph backend is added.

    Change-Id: I49c7b540caca18bada7166c9e7fb2f4b3475a5d3
    Closes-Bug: 1810440
    Signed-off-by: Wei Zhou <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: f/stein
Review: https://review.openstack.org/631263

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/631263
Committed: https://git.openstack.org/cgit/openstack/stx-config/commit/?id=603cc28734e17be92269832a7b041d414db9a331
Submitter: Zuul
Branch: f/stein

commit 3b4acbfc38f01f9e600d7fb41f0dbd55ba2166e4
Author: Wei Zhou <email address hidden>
Date: Fri Jan 11 15:26:20 2019 -0500

    /etc/ceph/ceph.client.None.keyring not created on standby controller

    This commit ensures that file /etc/ceph/ceph.client.None.keyring is
    created on both controllers after ceph backend is added.

    Change-Id: I49c7b540caca18bada7166c9e7fb2f4b3475a5d3
    Closes-Bug: 1810440
    Signed-off-by: Wei Zhou <email address hidden>

commit 1b8a344edd3c69c671ecce7c65a3c6822fff91f8
Author: Bart Wensley <email address hidden>
Date: Tue Jan 15 09:46:51 2019 -0600

    Ensure /etc/kubernetes/manifests directory exists

    Ensure the /etc/kubernetes/manifests directory exists before
    starting kubelet. This will avoid repeating kubelet logs on
    worker nodes, complaining that the directory does not exist.

    Also removing an unused yum config file for the kubernetes
    repo.

    Change-Id: Id63287481f9fe063f1b8c993d973a4b4517abd81
    Story: 2004520
    Task: 28908
    Signed-off-by: Bart Wensley <email address hidden>

commit 7dd943fe46681ff361533b65807416fc44adf971
Author: Eric MacDonald <email address hidden>
Date: Mon Jan 14 15:22:06 2019 -0500

    Fix mtce.pp to handle missing /etc/rmonfiles.d directory

    https://review.openstack.org/#/c/628687/ stopped packaging the
    query_ntp_servers.sh script. However, since there were no other
    files being packaged into that directory the spec file choose
    not to create an empty directory.

    When config controller called the mtce.pp manifest to install
    dynamic files into /etc/rmonfiles.d it could not. So it failed.

    This update adds a directory check block to the mtce.pp file
    to create the directoy if its not present.

    Testing: Install AIO SX in SM1

    Change-Id: Ib2dfadb261be6f9ebbaa7213eb6669b25158c779
    Closes-Bug: 1811693
    Signed-off-by: Eric MacDonald <email address hidden>

commit f44717154a78add0a0a2497048e03ab536dc615b
Author: Alex Kozyrev <email address hidden>
Date: Thu Jan 10 18:43:14 2019 -0500

    Add Barbican bootstrap and runtime manifests

    Barbican service is needed during bootstrap phase for StarlingX.
    Implement bootstrap and runtime manifests to achieve that.

    Change-Id: I6c22ebddacf8aec3a731f7f6d7a762f79f511c78
    Story: 2003108
    Task: 27700
    Signed-off-by: Alex Kozyrev <email address hidden>

commit f2a734a5f1d07603cde45dab675000688e87890a
Author: Joseph Richard <email address hidden>
Date: Tue Jan 8 11:41:07 2019 -0500

    Set bridge-nf-call-arptables

    This commit sets net.bridge.bridge-nf-call-arptables=1, which was
    found to be required for a containerized neutron deployment.

    Story: 2004520
    Task: 28789
    Change-Id: I0163535ed6bac7e775a63dfa3d54727c60f4208d
    Signed-off-by: Joseph Richard <email address hidden>

commit 8d14f3cd52392e1b3341f4d36a...