stx-openstack reapply failed after host-swact

[Peng Peng]

Brief Description
-----------------
After host swact success, tried to reapply stx-openstack, but the status stuck at "applying application manifest", and eventually apply-failed by operation aborted.

Severity
--------
Major

Steps to Reproduce
------------------
host-swact
application-apply
application-list

TC-name: z_containers/test_openstack_services.py::test_reapply_stx_openstack_no_change[controller-1]

Expected Behavior
------------------
reapply success

Actual Behavior
----------------
reapply failed

Reproducibility
---------------
Unknown - first time this is seen in sanity, will monitor

System Configuration
--------------------
Multi-node system

Lab-name: PV1

Branch/Pull Time/Commit
-----------------------
20200107T000000Z

Last Pass
---------
20191231T000000Z

Timestamp/Logs
--------------
[2020-01-07 10:56:41,078] 311 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.1:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-swact controller-0'

[2020-01-07 10:57:56,248] 311 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.1:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne application-list'
[2020-01-07 10:57:57,568] 433 DEBUG MainThread ssh.expect :: Output:
+---------------------+--------------------------------+-------------------------------+--------------------+---------+-----------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+--------------------------------+-------------------------------+--------------------+---------+-----------+
| platform-integ-apps | 1.0-8 | platform-integration-manifest | manifest.yaml | applied | completed |
| stx-openstack | 1.0-19-centos-stable-versioned | armada-manifest | stx-openstack.yaml | applied | completed |
+---------------------+--------------------------------+-------------------------------+--------------------+---------+-----------+

[2020-01-07 10:59:07,123] 476 DEBUG MainThread ssh.exec_cmd:: Executing command...
[2020-01-07 10:59:07,123] 311 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.1:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne application-apply stx-openstack'
[2020-01-07 10:59:08,437] 433 DEBUG MainThread ssh.expect :: Output:
+---------------+----------------------------------+
| Property | Value |
+---------------+----------------------------------+
| active | True |
| app_version | 1.0-19-centos-stable-versioned |
| created_at | 2020-01-07T08:45:41.812330+00:00 |
| manifest_file | stx-openstack.yaml |
| manifest_name | armada-manifest |
| name | stx-openstack |
| progress | None |
| status | applying |
| updated_at | 2020-01-07T10:55:12.836880+00:00 |
+---------------+----------------------------------+
Please use 'system application-list' or 'system application-show stx-openstack' to view the current progress.
controller-1:~$

system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.1:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne application-list'
[2020-01-07 10:59:09,802] 433 DEBUG MainThread ssh.expect :: Output:
+---------------------+--------------------------------+-------------------------------+--------------------+----------+----------------------------------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+--------------------------------+-------------------------------+--------------------+----------+----------------------------------+
| platform-integ-apps | 1.0-8 | platform-integration-manifest | manifest.yaml | applied | completed |
| stx-openstack | 1.0-19-centos-stable-versioned | armada-manifest | stx-openstack.yaml | applying | generating application overrides |
+---------------------+--------------------------------+-------------------------------+--------------------+----------+----------------------------------+

[2020-01-07 11:30:20,654] 311 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.1:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne application-list'
[2020-01-07 11:30:21,985] 433 DEBUG MainThread ssh.expect :: Output:
+---------------------+--------------------------------+-------------------------------+--------------------+----------+-------------------------------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+--------------------------------+-------------------------------+--------------------+----------+-------------------------------+
| platform-integ-apps | 1.0-8 | platform-integration-manifest | manifest.yaml | applied | completed |
| stx-openstack | 1.0-19-centos-stable-versioned | armada-manifest | stx-openstack.yaml | applying | applying application manifest |
+---------------------+--------------------------------+-------------------------------+--------------------+----------+-------------------------------+
controller-1:~$

[2020-01-07 11:30:37,101] 311 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.1:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne application-list'
[2020-01-07 11:30:38,367] 433 DEBUG MainThread ssh.expect :: Output:
+---------------------+--------------------------------+-------------------------------+--------------------+--------------+------------------------------------------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+--------------------------------+-------------------------------+--------------------+--------------+------------------------------------------+
| platform-integ-apps | 1.0-8 | platform-integration-manifest | manifest.yaml | applied | completed |
| stx-openstack | 1.0-19-centos-stable-versioned | armada-manifest | stx-openstack.yaml | apply-failed | operation aborted, check logs for detail |
+---------------------+--------------------------------+-------------------------------+--------------------+--------------+------------------------------------------+
controller-1:~$

Test Activity
-------------
Sanity

[Ghada Khalil (gkhalil)]

Assigning to containers PL for review/release recommendation

[Peng Peng (ppeng)]

Logs @
https://files.starlingx.kube.cengn.ca/launchpad/1858686

[Frank Miller (sensfan22)]

Assigning to Elena to investigate the logs and determine the cause of this failure.

[Austin Sun (sunausti)]

please re-test this case as openstack ussuri upgraded .

[Elena Taivan (etaivan)]

Brief description
After host swact, armada reports that openstack is failing due to a timeout for waiting ingress-sf44j pod.

Host swact was done
controller-0: /var/log/bash.log

2020-01-07T10:56:41.000 controller-0 -sh: info HISTORY: PID=1664749 UID=42425 system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.1:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-swact controller-0

SM logs indicate that all services were successfully brought up on controller-1
.
.
| 2020-01-07T10:57:24.813 | 564 | service-scn | postgres | enabling | enabled-active | enable success
| 2020-01-07T10:57:25.122 | 565 | service-scn | barbican-api | enabling-throttle | enabling | enabled-active state requested
| 2020-01-07T10:57:25.123 | 566 | service-scn | keystone | enabling-throttle | enabling | enabled-active state requested
| 2020-01-07T10:57:25.124 | 567 | service-scn | fm-mgr | enabling-throttle | enabling | enabled-active state requested
| 2020-01-07T10:57:25.124 | 568 | service-scn | sysinv-conductor | enabling-throttle | enabling | enabled-active state requested
| 2020-01-07T10:57:25.165 | 569 | service-scn | sysinv-conductor | enabling | enabled-active | enable success
| 2020-01-07T10:57:25.186 | 570 | service-scn | fm-mgr | enabling | enabled-active | enable success

STX-openstack is re-applied
controller-1 /var/log/bash.log

2020-01-07T10:59:07.000 controller-1 -sh: info HISTORY: PID=1316772 UID=42425 system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.1:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne application-apply stx-openstack

STX-openstack fails
sysinv.log

sysinv 2020-01-07 11:30:37.976 1313113 ERROR sysinv.conductor.kube_app [-] Failed to apply application manifest /manifests/stx-openstack/1.0-19-centos-stable-versioned/stx-openstack-stx-openstack.yaml. See /var/log/armada/stx-openstack-apply_2020-01-07-11-00-22.log for details.
sysinv 2020-01-07 11:30:37.977 1313113 INFO sysinv.conductor.kube_app [-] Exiting progress monitoring thread for app stx-openstack
sysinv 2020-01-07 11:30:38.171 1313113 ERROR sysinv.conductor.kube_app [-] Application apply aborted!.

armada logs
2020-01-07 11:00:36.880 16 INFO armada.handlers.wait [-] [chart=kube-system-ingress]: Waiting for resource type=pod, namespace=kube-system labels=release_group=osh-kube-system-ingress required=True for 1800s
.
.
2020-01-07 11:00:36.905 16 DEBUG armada.handlers.wait [-] [chart=kube-system-ingress]: pod ingress-sf44j not re...

[Frank Miller (sensfan22)]

Issues around ingress pod readiness were addressed earlier in 2020. Marking this LP as fix released.

If new issues with stx-openstack are seen after host-swacts please create a new LP as stx-openstack is now based on Ussuri and this LP was open early in 2020 when stx-openstack was based on Train.

[Peng Peng (ppeng)]

This issue was not reproduced on
Lab: WCP_69_70
Load: 2020-06-27_18-35-20