| 2019-06-21 12:43:39 |
Numan Waheed |
description |
Brief Description
-----------------
After lock/unlock standby controller, stx-openstack got reapply, but it is stuck at processing chart: osh-openstack-ceph-rgw.
Severity
--------
Major
Critical
Minor
Steps to Reproduce
------------------
lock/unlock standby controller
check stx-application reapply status
TC-name: test_lock_unlock_host.py::test_lock_unlock_host[controller]
Expected Behavior
------------------
Actual Behavior
----------------
Reproducibility
---------------
Intermittent
System Configuration
--------------------
Multi-node system
Lab-name: WCP_113-121
Branch/Pull Time/Commit
-----------------------
stx master as of 20190620T013000Z
Last Pass
---------
Lab: WCP_113_121
Load: 20190612T013000Z
Timestamp/Logs
--------------
[2019-06-20 17:32:08,722] 268 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.222.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne application-list'
[2019-06-20 17:32:10,214] 387 DEBUG MainThread ssh.expect :: Output:
+---------------------+------------------------------+-------------------------------+--------------------+---------------+----------------------------------------------------------------------------------------------------------------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+------------------------------+-------------------------------+--------------------+---------------+----------------------------------------------------------------------------------------------------------------+
| hello-kitty | 1.0 | hello-kitty | manifest.yaml | upload-failed | Upload of application hello-kitty (1.0) failed: Command '['helm-upload', 'starlingx', u'/scratch/apps/hello- |
| | | | | | kitty/1.0/charts/hello-kitty.tgz']' returned non-zero exit status 1 |
| | | | | | |
| platform-integ-apps | 1.0-7 | platform-integration-manifest | manifest.yaml | applied | completed |
| stx-openstack | 1.0-16-centos-stable- | armada-manifest | stx-openstack.yaml | applied | completed
[2019-06-20 17:32:10,672] 268 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.222.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-unlock controller-0'
[2019-06-20 17:38:11,919] 268 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.222.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-show controller-0'
[2019-06-20 17:38:13,488] 387 DEBUG MainThread ssh.expect :: Output:
+---------------------+-----------------------------------------------------------------------+
| Property | Value |
+---------------------+-----------------------------------------------------------------------+
| action | none |
| administrative | unlocked |
| availability | available |
[2019-06-20 17:38:19,192] 268 DEBUG MainThread ssh.send :: Send 'kubectl get nodes'
[2019-06-20 17:38:19,417] 387 DEBUG MainThread ssh.expect :: Output:
NAME STATUS ROLES AGE VERSION
compute-0 Ready <none> 165m v1.13.5
compute-1 Ready <none> 165m v1.13.5
compute-2 Ready <none> 165m v1.13.5
compute-3 Ready <none> 165m v1.13.5
compute-4 Ready <none> 165m v1.13.5
controller-0 Ready master 3h54m v1.13.5
controller-1 Ready master 3h7m v1.13.5
[sysadmin@controller-1 ~(keystone_admin)]$
[2019-06-20 17:38:19,417] 268 DEBUG MainThread ssh.send :: Send 'echo $?'
[2019-06-20 17:38:19,520] 387 DEBUG MainThread ssh.expect :: Output:
0
[sysadmin@controller-1 ~(keystone_admin)]$
[2019-06-20 17:38:19,521] 591 INFO MainThread kube_helper.wait_for_nodes_ready:: All nodes are ready: controller-0
[2019-06-20 17:38:19,521] 212 INFO MainThread container_helper.wait_for_apps_status:: Wait for ['stx-openstack'] application(s) to reach status: applied
[2019-06-20 17:38:19,521] 423 DEBUG MainThread ssh.exec_cmd:: Executing command...
[2019-06-20 17:38:19,521] 268 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.222.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne application-list'
[2019-06-20 17:38:21,004] 387 DEBUG MainThread ssh.expect :: Output:
+---------------------+------------------------------+-------------------------------+----------------+---------------+------------------------------------------------------------------------------------------------------------------------------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+------------------------------+-------------------------------+----------------+---------------+------------------------------------------------------------------------------------------------------------------------------+
| hello-kitty | 1.0 | hello-kitty | manifest.yaml | upload-failed | Upload of application hello-kitty (1.0) failed: Command '['helm-upload', 'starlingx', u'/scratch/apps/hello-kitty/1.0/charts |
| | | | | | /hello-kitty.tgz']' returned non-zero exit status 1 |
| | | | | | |
| platform-integ-apps | 1.0-7 | platform-integration-manifest | manifest.yaml | applied | completed |
| stx-openstack | 1.0-16-centos-stable- | armada-manifest | stx-openstack. | applying | processing chart: osh-kube-system-ingress, overall completion: 4.0% |
| | versioned | | yaml | |
[2019-06-20 17:44:09,192] 268 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.222.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne application-list'
[2019-06-20 17:44:10,754] 387 DEBUG MainThread ssh.expect :: Output:
+---------------------+------------------------------+-------------------------------+----------------+---------------+------------------------------------------------------------------------------------------------------------------------------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+------------------------------+-------------------------------+----------------+---------------+------------------------------------------------------------------------------------------------------------------------------+
| hello-kitty | 1.0 | hello-kitty | manifest.yaml | upload-failed | Upload of application hello-kitty (1.0) failed: Command '['helm-upload', 'starlingx', u'/scratch/apps/hello-kitty/1.0/charts |
| | | | | | /hello-kitty.tgz']' returned non-zero exit status 1 |
| | | | | | |
| platform-integ-apps | 1.0-7 | platform-integration-manifest | manifest.yaml | applied | completed |
| stx-openstack | 1.0-16-centos-stable- | armada-manifest | stx-openstack. | applying | processing chart: osh-openstack-ceph-rgw, overall completion: 44.0% |
Test Activity
-------------
Sanity |
Brief Description
-----------------
After lock/unlock standby controller, stx-openstack got reapply, but it is stuck at processing chart: osh-openstack-ceph-rgw.
Severity
--------
Major
Steps to Reproduce
------------------
lock/unlock standby controller
check stx-application reapply status
TC-name: test_lock_unlock_host.py::test_lock_unlock_host[controller]
Expected Behavior
------------------
Actual Behavior
----------------
Reproducibility
---------------
Intermittent
System Configuration
--------------------
Multi-node system
Lab-name: WCP_113-121
Branch/Pull Time/Commit
-----------------------
stx master as of 20190620T013000Z
Last Pass
---------
Lab: WCP_113_121
Load: 20190612T013000Z
Timestamp/Logs
--------------
[2019-06-20 17:32:08,722] 268 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.222.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne application-list'
[2019-06-20 17:32:10,214] 387 DEBUG MainThread ssh.expect :: Output:
+---------------------+------------------------------+-------------------------------+--------------------+---------------+----------------------------------------------------------------------------------------------------------------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+------------------------------+-------------------------------+--------------------+---------------+----------------------------------------------------------------------------------------------------------------+
| hello-kitty | 1.0 | hello-kitty | manifest.yaml | upload-failed | Upload of application hello-kitty (1.0) failed: Command '['helm-upload', 'starlingx', u'/scratch/apps/hello- |
| | | | | | kitty/1.0/charts/hello-kitty.tgz']' returned non-zero exit status 1 |
| | | | | | |
| platform-integ-apps | 1.0-7 | platform-integration-manifest | manifest.yaml | applied | completed |
| stx-openstack | 1.0-16-centos-stable- | armada-manifest | stx-openstack.yaml | applied | completed
[2019-06-20 17:32:10,672] 268 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.222.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-unlock controller-0'
[2019-06-20 17:38:11,919] 268 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.222.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-show controller-0'
[2019-06-20 17:38:13,488] 387 DEBUG MainThread ssh.expect :: Output:
+---------------------+-----------------------------------------------------------------------+
| Property | Value |
+---------------------+-----------------------------------------------------------------------+
| action | none |
| administrative | unlocked |
| availability | available |
[2019-06-20 17:38:19,192] 268 DEBUG MainThread ssh.send :: Send 'kubectl get nodes'
[2019-06-20 17:38:19,417] 387 DEBUG MainThread ssh.expect :: Output:
NAME STATUS ROLES AGE VERSION
compute-0 Ready <none> 165m v1.13.5
compute-1 Ready <none> 165m v1.13.5
compute-2 Ready <none> 165m v1.13.5
compute-3 Ready <none> 165m v1.13.5
compute-4 Ready <none> 165m v1.13.5
controller-0 Ready master 3h54m v1.13.5
controller-1 Ready master 3h7m v1.13.5
[sysadmin@controller-1 ~(keystone_admin)]$
[2019-06-20 17:38:19,417] 268 DEBUG MainThread ssh.send :: Send 'echo $?'
[2019-06-20 17:38:19,520] 387 DEBUG MainThread ssh.expect :: Output:
0
[sysadmin@controller-1 ~(keystone_admin)]$
[2019-06-20 17:38:19,521] 591 INFO MainThread kube_helper.wait_for_nodes_ready:: All nodes are ready: controller-0
[2019-06-20 17:38:19,521] 212 INFO MainThread container_helper.wait_for_apps_status:: Wait for ['stx-openstack'] application(s) to reach status: applied
[2019-06-20 17:38:19,521] 423 DEBUG MainThread ssh.exec_cmd:: Executing command...
[2019-06-20 17:38:19,521] 268 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.222.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne application-list'
[2019-06-20 17:38:21,004] 387 DEBUG MainThread ssh.expect :: Output:
+---------------------+------------------------------+-------------------------------+----------------+---------------+------------------------------------------------------------------------------------------------------------------------------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+------------------------------+-------------------------------+----------------+---------------+------------------------------------------------------------------------------------------------------------------------------+
| hello-kitty | 1.0 | hello-kitty | manifest.yaml | upload-failed | Upload of application hello-kitty (1.0) failed: Command '['helm-upload', 'starlingx', u'/scratch/apps/hello-kitty/1.0/charts |
| | | | | | /hello-kitty.tgz']' returned non-zero exit status 1 |
| | | | | | |
| platform-integ-apps | 1.0-7 | platform-integration-manifest | manifest.yaml | applied | completed |
| stx-openstack | 1.0-16-centos-stable- | armada-manifest | stx-openstack. | applying | processing chart: osh-kube-system-ingress, overall completion: 4.0% |
| | versioned | | yaml | |
[2019-06-20 17:44:09,192] 268 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.222.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne application-list'
[2019-06-20 17:44:10,754] 387 DEBUG MainThread ssh.expect :: Output:
+---------------------+------------------------------+-------------------------------+----------------+---------------+------------------------------------------------------------------------------------------------------------------------------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+------------------------------+-------------------------------+----------------+---------------+------------------------------------------------------------------------------------------------------------------------------+
| hello-kitty | 1.0 | hello-kitty | manifest.yaml | upload-failed | Upload of application hello-kitty (1.0) failed: Command '['helm-upload', 'starlingx', u'/scratch/apps/hello-kitty/1.0/charts |
| | | | | | /hello-kitty.tgz']' returned non-zero exit status 1 |
| | | | | | |
| platform-integ-apps | 1.0-7 | platform-integration-manifest | manifest.yaml | applied | completed |
| stx-openstack | 1.0-16-centos-stable- | armada-manifest | stx-openstack. | applying | processing chart: osh-openstack-ceph-rgw, overall completion: 44.0% |
Test Activity
-------------
Sanity |
|
