Upgrade: Data migration fails while upgrading controller-1

[Carmen Rata]

Brief Description
-----------------
During an upgrade procedure, while upgrading N+1 controller node, data migration fails.

Severity
--------
Major

Steps to Reproduce
------------------
Install AIO-DX system in DC System Controller IPv6 configuration.
system upgrade-start
system Lock controller-1
system host-upgrade controller-1

Expected Behavior
------------------
After "system host-upgrade controller-1", upgrade status shows "data-migration completed".

Actual Behavior
----------------
Upgrade status shows "data-migration failed"

Reproducibility
---------------
100%

System Configuration
--------------------
Seen on AIO-DX systems with DC System Controller configuration.

Branch/Pull Time/Commit
-----------------------

Last Pass
---------
Unknown

Timestamp/Logs
--------------
2020-06-04T19:16:44.631 localhost controller_config[5691]: info sysinv 2020-06-04 19:16:44.233 37218 ERROR sysinv.puppet.puppet [-] failed to create secure_system config: MaxRetryError: HTTPSConnectionPool(host='abcd:205::1', port=6443): Max retries exceeded with url: /api/v1/namespaces/dc-cert/secrets/dc-adminep-root-ca-certificate (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fce228a1790>: Failed to establish a new connection: [Errno 101] ENETUNREACH',))

Test Activity
-------------
Developer Testing

Workaround
----------
Copy "/etc/kubernates/admin.conf" to "admin-test.conf". Modify admin-test.conf, as follows:
- change "server: https://[<cluster IPaddr>]:6443" to "server: https://[<management IPaddr>]:6443"

After "system upgrade-start" and before "system host-upgrade controller-1" do this:
sudo cp /etc/kubernetes/admin-test.conf /opt/platform/config/20.06/kubernetes/admin.conf

[Ghada Khalil (gkhalil)]

stx.4.0 / high priority - issue affects the stx.4.0 system upgrades feature

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/736193

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/736193
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=002a818b1950df125ab517ea4e62f1b7b5d59eca
Submitter: Zuul
Branch: master

commit 002a818b1950df125ab517ea4e62f1b7b5d59eca
Author: Carmen Rata <email address hidden>
Date: Wed Jun 17 08:28:20 2020 -0400

    Fix data-migration failure when upgrading N+1 node

    When upgrading to N+1 node, cluster IP address from admin.conf
    needs to be replaced with the floating management IP address.
    This is to prevent failure in data migration when kubernetes api
    accesses dc-adminep-root-ca-certificate.
    The update of cluster IP address is temporary and done on a
    copy of the original /etc/kubernetes/admin.conf.
    This commit applies changes from the N side.

    Closes-Bug: 1883763

    Change-Id: Ibc2dbbab5016d9c58248ebb5b0fb52299e99705c
    Signed-off-by: Carmen Rata <email address hidden>