Brief Description ----------------- After lock/unlock standby controller, stx-openstack got reapply, but it is stuck at processing chart: osh-openstack-ceph-rgw. Severity -------- Major Critical Minor Steps to Reproduce ------------------ lock/unlock standby controller check stx-application reapply status TC-name: test_lock_unlock_host.py::test_lock_unlock_host[controller] Expected Behavior ------------------ Actual Behavior ---------------- Reproducibility --------------- Intermittent System Configuration -------------------- Multi-node system Lab-name: WCP_113-121 Branch/Pull Time/Commit ----------------------- stx master as of 20190620T013000Z Last Pass --------- Lab: WCP_113_121 Load: 20190612T013000Z Timestamp/Logs -------------- [2019-06-20 17:32:08,722] 268 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.222.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne application-list' [2019-06-20 17:32:10,214] 387 DEBUG MainThread ssh.expect :: Output: +---------------------+------------------------------+-------------------------------+--------------------+---------------+----------------------------------------------------------------------------------------------------------------+ | application | version | manifest name | manifest file | status | progress | +---------------------+------------------------------+-------------------------------+--------------------+---------------+----------------------------------------------------------------------------------------------------------------+ | hello-kitty | 1.0 | hello-kitty | manifest.yaml | upload-failed | Upload of application hello-kitty (1.0) failed: Command '['helm-upload', 'starlingx', u'/scratch/apps/hello- | | | | | | | kitty/1.0/charts/hello-kitty.tgz']' returned non-zero exit status 1 | | | | | | | | | platform-integ-apps | 1.0-7 | platform-integration-manifest | manifest.yaml | applied | completed | | stx-openstack | 1.0-16-centos-stable- | armada-manifest | stx-openstack.yaml | applied | completed [2019-06-20 17:32:10,672] 268 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.222.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-unlock controller-0' [2019-06-20 17:38:11,919] 268 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.222.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-show controller-0' [2019-06-20 17:38:13,488] 387 DEBUG MainThread ssh.expect :: Output: +---------------------+-----------------------------------------------------------------------+ | Property | Value | +---------------------+-----------------------------------------------------------------------+ | action | none | | administrative | unlocked | | availability | available | [2019-06-20 17:38:19,192] 268 DEBUG MainThread ssh.send :: Send 'kubectl get nodes' [2019-06-20 17:38:19,417] 387 DEBUG MainThread ssh.expect :: Output: NAME STATUS ROLES AGE VERSION compute-0 Ready 165m v1.13.5 compute-1 Ready 165m v1.13.5 compute-2 Ready 165m v1.13.5 compute-3 Ready 165m v1.13.5 compute-4 Ready 165m v1.13.5 controller-0 Ready master 3h54m v1.13.5 controller-1 Ready master 3h7m v1.13.5 [sysadmin@controller-1 ~(keystone_admin)]$ [2019-06-20 17:38:19,417] 268 DEBUG MainThread ssh.send :: Send 'echo $?' [2019-06-20 17:38:19,520] 387 DEBUG MainThread ssh.expect :: Output: 0 [sysadmin@controller-1 ~(keystone_admin)]$ [2019-06-20 17:38:19,521] 591 INFO MainThread kube_helper.wait_for_nodes_ready:: All nodes are ready: controller-0 [2019-06-20 17:38:19,521] 212 INFO MainThread container_helper.wait_for_apps_status:: Wait for ['stx-openstack'] application(s) to reach status: applied [2019-06-20 17:38:19,521] 423 DEBUG MainThread ssh.exec_cmd:: Executing command... [2019-06-20 17:38:19,521] 268 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.222.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne application-list' [2019-06-20 17:38:21,004] 387 DEBUG MainThread ssh.expect :: Output: +---------------------+------------------------------+-------------------------------+----------------+---------------+------------------------------------------------------------------------------------------------------------------------------+ | application | version | manifest name | manifest file | status | progress | +---------------------+------------------------------+-------------------------------+----------------+---------------+------------------------------------------------------------------------------------------------------------------------------+ | hello-kitty | 1.0 | hello-kitty | manifest.yaml | upload-failed | Upload of application hello-kitty (1.0) failed: Command '['helm-upload', 'starlingx', u'/scratch/apps/hello-kitty/1.0/charts | | | | | | | /hello-kitty.tgz']' returned non-zero exit status 1 | | | | | | | | | platform-integ-apps | 1.0-7 | platform-integration-manifest | manifest.yaml | applied | completed | | stx-openstack | 1.0-16-centos-stable- | armada-manifest | stx-openstack. | applying | processing chart: osh-kube-system-ingress, overall completion: 4.0% | | | versioned | | yaml | | [2019-06-20 17:44:09,192] 268 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.222.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne application-list' [2019-06-20 17:44:10,754] 387 DEBUG MainThread ssh.expect :: Output: +---------------------+------------------------------+-------------------------------+----------------+---------------+------------------------------------------------------------------------------------------------------------------------------+ | application | version | manifest name | manifest file | status | progress | +---------------------+------------------------------+-------------------------------+----------------+---------------+------------------------------------------------------------------------------------------------------------------------------+ | hello-kitty | 1.0 | hello-kitty | manifest.yaml | upload-failed | Upload of application hello-kitty (1.0) failed: Command '['helm-upload', 'starlingx', u'/scratch/apps/hello-kitty/1.0/charts | | | | | | | /hello-kitty.tgz']' returned non-zero exit status 1 | | | | | | | | | platform-integ-apps | 1.0-7 | platform-integration-manifest | manifest.yaml | applied | completed | | stx-openstack | 1.0-16-centos-stable- | armada-manifest | stx-openstack. | applying | processing chart: osh-openstack-ceph-rgw, overall completion: 44.0% | Test Activity ------------- Sanity