stx-monitor stuck at applying status when apply is not possible - it should reach apply-failed
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
StarlingX |
Fix Released
|
Medium
|
Kevin Smith |
Bug Description
Brief Description
-----------------
stx-monitor app applied and deleted successfully on SX system. After Backup and Restore, tried to apply stx-monitor on system, but app stucked at 'applyin' status. And host host cannot be unlocked after locking due to this issue
Severity
--------
Major
Steps to Reproduce
------------------
BnR on SX system
check stx-monitor app status
TC-name: sanity after BnR
Expected Behavior
------------------
stx-monitor applied
Actual Behavior
----------------
stx-monitor stuck as applying
Reproducibility
---------------
Unknown - first time this is seen in sanity, will monitor
System Configuration
-------
One node system
Lab-name: wcp-112
Branch/Pull Time/Commit
-------
2020-03-09_04-10-00
Last Pass
---------
unknown
Timestamp/Logs
--------------
[2020-03-10 07:57:07,891] 314 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://[abcd:204:
[2020-03-10 07:57:09,050] 436 DEBUG MainThread ssh.expect :: Output:
+------
| application | version | manifest name | manifest file | status | progress |
+------
| oidc-auth-apps | 1.0-0 | oidc-auth-manifest | manifest.yaml | uploaded | completed |
| platform-integ-apps | 1.0-8 | platform-
| stx-monitor | 1.0-1 | monitor-
+------
[2020-03-10 07:58:40,715] 314 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://[abcd:204:
[2020-03-10 07:58:43,088] 436 DEBUG MainThread ssh.expect :: Output:
Application stx-monitor deleted.
controller-0:~$
BnR ....
[2020-03-10 20:06:27,015] 314 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://[abcd:204:
[2020-03-10 20:06:28,164] 436 DEBUG MainThread ssh.expect :: Output:
+------
| application | version | manifest name | manifest file | status | progress |
+------
| oidc-auth-apps | 1.0-0 | oidc-auth-manifest | manifest.yaml | uploaded | completed |
| platform-integ-apps | 1.0-8 | platform-
| stx-monitor | 1.0-1 | monitor-
+------
controller-0:~$
[2020-03-10 20:06:28,164] 314 DEBUG MainThread ssh.send :: Send 'echo $?'
[2020-03-10 20:06:28,267] 436 DEBUG MainThread ssh.expect :: Output:
0
controller-0:~$
[2020-03-10 20:06:28,268] 254 INFO MainThread container_
[2020-03-10 20:06:28,268] 144 INFO MainThread container_
[2020-03-10 20:06:28,268] 287 INFO MainThread test_stx_
[2020-03-10 20:06:28,268] 296 INFO MainThread container_
[2020-03-10 20:06:28,269] 1604 DEBUG MainThread ssh.get_
[2020-03-10 20:06:28,269] 479 DEBUG MainThread ssh.exec_cmd:: Executing command...
[2020-03-10 20:06:28,269] 314 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://[abcd:204:
[2020-03-10 21:05:45,153] 314 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://[abcd:204:
[2020-03-10 21:05:46,275] 436 DEBUG MainThread ssh.expect :: Output:
+------
| application | version | manifest name | manifest file | status | progress |
+------
| oidc-auth-apps | 1.0-0 | oidc-auth-manifest | manifest.yaml | uploaded | completed |
| platform-integ-apps | 1.0-8 | platform-
| stx-monitor | 1.0-1 | monitor-
+------
system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://[abcd:204:
Test Activity
-------------
Sanity
tags: | added: stx.retestneeded |
Changed in starlingx: | |
assignee: | nobody → Mihnea Saracin (msaracin) |
It seems, after BnR, platform- deployment- manager pod is not running.
[root@controller-0 sysadmin( keystone_ admin)] # kubectl get pods --all-namespaces kube-controller s-855577b7b5- c49fs 1/1 Running 6889846b6b- fw7qh 1/1 Running controller- 0 1/1 Running -manager- controller- 0 1/1 Running ds-amd64- x6vmc 1/1 Running controller- 0 1/1 Running cni-ds- amd64-r67q5 1/1 Running deploy- d6b59fcb- ldb9p 1/1 Running keystone_ admin)] #
NAMESPACE NAME READY STATUS
RESTARTS AGE
kube-system calico-
7 24h
kube-system calico-node-kg5jv 1/1 Running
3 24h
kube-system coredns-
3 24h
kube-system kube-apiserver-
3 24h
kube-system kube-controller
4 24h
kube-system kube-multus-
3 24h
kube-system kube-proxy-9flbc 1/1 Running
3 24h
kube-system kube-scheduler-
4 24h
kube-system kube-sriov-
3 24h
kube-system tiller-
3 24h
[root@controller-0 sysadmin(
[root@controller-0 sysadmin( keystone_ admin)] # KUBECONFIG= /etc/kubernetes /admin. conf /bin/kubectl apply -f andy_backup/ deployment- config. yaml deployment unchanged platform- certificate unchanged system- endpoint configured system- license unchanged deployment- config. yaml": no matches for kind "System" in version "starlingx. windriver. com/v1" deployment- config. yaml": no matches for kind "DataNetwork" in version "starlingx. windriver. com/v1" deployment- config. yaml": no matches for kind "HostProfile" in version "starlingx. windriver. com/v1" deployment- config. yaml": no matches for kind "Host" in version "starlingx. windriver. com/v1"
namespace/
secret/
secret/
secret/
unable to recognize "andy_backup/
unable to recognize "andy_backup/
unable to recognize "andy_backup/
unable to recognize "andy_backup/