FM and dcmanager databases not migrated during upgrade to stx.4.0

[Bart Wensley]

Brief Description
-----------------
When a system is upgraded to stx.4.0 some databases are not preserved:
- The contents of the FM database are not preserved (except in AIO-SX) so all event logs and alarm suppression configuration are lost.
- The dcmanager and dcorch databases are preserved, but their schemas are not migrated. This causes dcmanager to fail after controller-1 is upgraded to stx.4.0.

Severity
--------
Critical: upgrades to stx.4.0 fail for distributed cloud systems

Steps to Reproduce
------------------
Upgrade a distributed cloud system to stx.4.0

Expected Behavior
------------------
All FM data is migrated (except alarm table) and dcmanager/dcorch database schemas are migrated.

Actual Behavior
----------------
No FM data is migrated and dcmanager/dcorch database schemas are not migrated.

Reproducibility
---------------
Reproducible

System Configuration
--------------------
All, although the impact to distributed cloud configurations is more severe.

Branch/Pull Time/Commit
-----------------------
BUILD_ID="2020-06-26_01-17-51"

Last Pass
---------
Never

Timestamp/Logs
--------------
There are various failure logs in dcmanager.log but also the dcmanager CLI fails as follows:

$ dcmanager subcloud list
ERROR (app)

Test Activity
-------------
Developer Testing

Workaround
----------
No workaround for the FM data. For the dcmanager/dcorch issue, after controller-1 has been upgraded and becomes active, run these commands as root on controller-1:

sudo dcmanager-manager db_sync

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/738284

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/738284
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=2a6d43d669d5219050b64007b761d33af23742bd
Submitter: Zuul
Branch: master

commit 2a6d43d669d5219050b64007b761d33af23742bd
Author: John Kung <email address hidden>
Date: Fri Jun 26 17:09:22 2020 -0400

    Migrate dcmanager database on upgrade

    Perform the database migration for dcmanager database on upgrade
    by setting up configuration file and performing the
    database migration.

    Tests:
    Perform upgrade and verify upgrade database migrations
    Verify ability to add data_install data via subcloud update

    Change-Id: Ia9079b27e77aea1924ca13d7a2c586b90293948d
    Partial-bug: 1885328
    Signed-off-by: John Kung <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/738655

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/738655
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=85eb7a8415ac2356144f6e40663335c4ec0ff1c0
Submitter: Zuul
Branch: master

commit 85eb7a8415ac2356144f6e40663335c4ec0ff1c0
Author: John Kung <email address hidden>
Date: Tue Jun 30 11:55:44 2020 -0400

    Migrate fm, dcorch databases on upgrade

    Perform the database migration on platform upgrade for fm and,
    on SystemController, dcorch databases. The databases are
    migrated on from side, and on N+1 side setting up configuration
    file and performing the database migration.

    Skip the fm 'alarm' table in case there were non-management
    affecting alarms prior to the upgrade, that do not need to
    appear again after controller-1 is upgraded.

    When added to prior config commit 2a6d43d669,
    this closes bug 1885328

    Tests:
    Perform platform upgrade on SystemController
    Verify upgrade database migrations for fm
    Verify upgrade database migrations for dcorch

    Change-Id: I1a3325c019f5d8fd4bd39f99d61b7610132c2d4f
    Partial-bug: 1885328
    Signed-off-by: John Kung <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/741058

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/741058
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=fc5690e73f38c473d5871f284249e22a562f44cc
Submitter: Zuul
Branch: master

commit fc5690e73f38c473d5871f284249e22a562f44cc
Author: David Sullivan <email address hidden>
Date: Tue Jul 14 17:27:21 2020 -0400

    Backup fm.conf during simplex upgrade

    We use a temporary fm.conf during the data migration. We need to backup
    and restore the system fm.conf for the fm service to come up after data
    migration.

    Change-Id: I65a6ec74782174d322b7fa7f8a3bb47c1bc3db09
    Closes-Bug: 1885328
    Signed-off-by: David Sullivan <email address hidden>

[Ghada Khalil (gkhalil)]

@David, please cherrypick the additional fix above to the r/stx.4.0 branch

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: r/stx.4.0
Review: https://review.opendev.org/741457

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/741457
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=65fd34940c047691f9e6458655d7bc6a7f4c6164
Submitter: Zuul
Branch: r/stx.4.0

commit 65fd34940c047691f9e6458655d7bc6a7f4c6164
Author: David Sullivan <email address hidden>
Date: Tue Jul 14 17:27:21 2020 -0400

    Backup fm.conf during simplex upgrade

    We use a temporary fm.conf during the data migration. We need to backup
    and restore the system fm.conf for the fm service to come up after data
    migration.

    Change-Id: I65a6ec74782174d322b7fa7f8a3bb47c1bc3db09
    Closes-Bug: 1885328
    Signed-off-by: David Sullivan <email address hidden>