StarlingX

stx-admin-ks-user/keystone bootstrap is not is not ready on AIO-SX

Bug #1929775 reported by Yvonne Ding
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
Low
Yvonne Ding

Bug Description

Brief Description
-----------------
Job keystone-bootstrap and stx-admin-ks-user is not ready on AIO-SX. stx-openstack apply is failed when vim_progress_status is services-disabled after unlock controller.

Severity
------------------------------
Minor

Steps to Reproduce
-----------------------------
1. install starlingx master load on AIO-SX
2. run openstack mtc regression
3. lock/unlock controller
4. system application-apply stx-openstack when vim_progress_status is services-disabled

Expected Behavior
------------------
stx-openstack applied successfully

Actual Behavior
----------------
stx-openstack apply failed

Reproducibility
----------------------------
Reproducible

System Configuration
----------------------------
AIO-SX

Branch/Pull Time/Commit
-----------------------
master:
OS="centos"
SW_VERSION="20.12"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20210429T040444Z"

stx/5.0
OS="centos"
SW_VERSION="21.05"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="r/stx.5.0"
BUILD_DATE="2021-04-21 23:03:24 +0000"

Timestamp/Logs
----------------------------
2021-03-05 16:25:42.977 627 ERROR armada.handlers.wait [-] [chart=openstack-keystone]: Timed out waiting for jobs (namespace=openstack, labels=(release_group=osh-openstack-keystone)). These jobs were not ready=['keystone-bootstrap', 'stx-admin-ks-user']^[[00m
2021-03-05 16:25:42.978 627 ERROR armada.handlers.armada [-] Chart deploy [openstack-keystone] failed: armada.exceptions.k8s_exceptions.KubernetesWatchTimeoutException: Timed out waiting for jobs (namespace=openstack, labels=(release_group=osh-openstack-keystone)). These jobs were not ready=['keystone-bootstrap', 'stx-admin-ks-user']
2021-03-05 16:25:42.978 627 ERROR armada.handlers.armada Traceback (most recent call last):
2021-03-05 16:25:42.978 627 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 225, in handle_result
2021-03-05 16:25:42.978 627 ERROR armada.handlers.armada result = get_result()
2021-03-05 16:25:42.978 627 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 236, in <lambda>
2021-03-05 16:25:42.978 627 ERROR armada.handlers.armada if (handle_result(chart, lambda: deploy_chart(chart))):
2021-03-05 16:25:42.978 627 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 214, in deploy_chart
2021-03-05 16:25:42.978 627 ERROR armada.handlers.armada chart, cg_test_all_charts, prefix, known_releases)
2021-03-05 16:25:42.978 627 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/chart_deploy.py", line 248, in execute
2021-03-05 16:25:42.978 627 ERROR armada.handlers.armada chart_wait.wait(timer)
2021-03-05 16:25:42.978 627 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py", line 134, in wait
2021-03-05 16:25:42.978 627 ERROR armada.handlers.armada wait.wait(timeout=timeout)
2021-03-05 16:25:42.978 627 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py", line 294, in wait
2021-03-05 16:25:42.978 627 ERROR armada.handlers.armada modified = self._wait(deadline)
2021-03-05 16:25:42.978 627 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py", line 354, in _wait
2021-03-05 16:25:42.978 627 ERROR armada.handlers.armada raise k8s_exceptions.KubernetesWatchTimeoutException(error)
2021-03-05 16:25:42.978 627 ERROR armada.handlers.armada armada.exceptions.k8s_exceptions.KubernetesWatchTimeoutException: Timed out waiting for jobs (namespace=openstack, labels=(release_group=osh-openstack-keystone)). These jobs were not ready=['keystone-bootstrap', 'stx-admin-ks-user']
2021-03-05 16:25:42.978 627 ERROR armada.handlers.armada ^[[00m
2021-03-05 16:25:42.979 627 ERROR armada.handlers.armada [-] Chart deploy(s) failed: ['openstack-keystone']^[[00m
2021-03-05 16:25:43.817 627 INFO armada.handlers.lock [-] Releasing lock^[[00m
2021-03-05 16:25:43.841 627 ERROR armada.cli [-] Caught internal exception: armada.exceptions.armada_exceptions.ChartDeployException: Exception deploying charts: ['openstack-keystone']
2021-03-05 16:25:43.841 627 ERROR armada.cli Traceback (most recent call last):
2021-03-05 16:25:43.841 627 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/_init_.py", line 38, in safe_invoke
2021-03-05 16:25:43.841 627 ERROR armada.cli self.invoke()
2021-03-05 16:25:43.841 627 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 213, in invoke
2021-03-05 16:25:43.841 627 ERROR armada.cli resp = self.handle(documents, tiller)
2021-03-05 16:25:43.841 627 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py", line 81, in func_wrapper
2021-03-05 16:25:43.841 627 ERROR armada.cli return future.result()
2021-03-05 16:25:43.841 627 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
2021-03-05 16:25:43.841 627 ERROR armada.cli return self.__get_result()
2021-03-05 16:25:43.841 627 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2021-03-05 16:25:43.841 627 ERROR armada.cli raise self._exception
2021-03-05 16:25:43.841 627 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
2021-03-05 16:25:43.841 627 ERROR armada.cli result = self.fn(*self.args, **self.kwargs)
2021-03-05 16:25:43.841 627 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 256, in handle
2021-03-05 16:25:43.841 627 ERROR armada.cli return armada.sync()
2021-03-05 16:25:43.841 627 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 252, in sync
2021-03-05 16:25:43.841 627 ERROR armada.cli raise armada_exceptions.ChartDeployException(failures)
2021-03-05 16:25:43.841 627 ERROR armada.cli armada.exceptions.armada_exceptions.ChartDeployException: Exception deploying charts: ['openstack-keystone']
2021-03-05 16:25:43.841 627 ERROR armada.cli ^[[00m

2021-05-18 20:12:56.325 68 ERROR armada.handlers.wait [-] [chart=openstack-keystone]: Timed out waiting for jobs (namespace=openstack, labels=(release_group=osh-openstack-keystone)). These jobs were not ready=['keystone-bootstrap', 'keystone-domain-manage', 'stx-admin-ks-user']
2021-05-18 20:12:56.326 68 ERROR armada.handlers.armada [-] Chart deploy [openstack-keystone] failed: armada.exceptions.k8s_exceptions.KubernetesWatchTimeoutException: Timed out waiting for jobs (namespace=openstack, labels=(release_group=osh-openstack-keystone)). These jobs were not ready=['keystone-bootstrap', 'keystone-domain-manage', 'stx-admin-ks-user']
2021-05-18 20:12:56.326 68 ERROR armada.handlers.armada Traceback (most recent call last):
2021-05-18 20:12:56.326 68 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 225, in handle_result
2021-05-18 20:12:56.326 68 ERROR armada.handlers.armada result = get_result()
2021-05-18 20:12:56.326 68 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 236, in <lambda>
2021-05-18 20:12:56.326 68 ERROR armada.handlers.armada if (handle_result(chart, lambda: deploy_chart(chart))):
2021-05-18 20:12:56.326 68 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 214, in deploy_chart
2021-05-18 20:12:56.326 68 ERROR armada.handlers.armada chart, cg_test_all_charts, prefix, known_releases)
2021-05-18 20:12:56.326 68 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/chart_deploy.py", line 248, in execute
2021-05-18 20:12:56.326 68 ERROR armada.handlers.armada chart_wait.wait(timer)
2021-05-18 20:12:56.326 68 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py", line 134, in wait
2021-05-18 20:12:56.326 68 ERROR armada.handlers.armada wait.wait(timeout=timeout)
2021-05-18 20:12:56.326 68 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py", line 294, in wait
2021-05-18 20:12:56.326 68 ERROR armada.handlers.armada modified = self._wait(deadline)
2021-05-18 20:12:56.326 68 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py", line 354, in _wait
2021-05-18 20:12:56.326 68 ERROR armada.handlers.armada raise k8s_exceptions.KubernetesWatchTimeoutException(error)
2021-05-18 20:12:56.326 68 ERROR armada.handlers.armada armada.exceptions.k8s_exceptions.KubernetesWatchTimeoutException: Timed out waiting for jobs (namespace=openstack, labels=(release_group=osh-openstack-keystone)). These jobs were not ready=['keystone-bootstrap', 'keystone-domain-manage', 'stx-admin-ks-user']
2021-05-18 20:12:56.326 68 ERROR armada.handlers.armada
2021-05-18 20:12:56.328 68 ERROR armada.handlers.armada [-] Chart deploy(s) failed: ['openstack-keystone']
2021-05-18 20:12:57.141 68 INFO armada.handlers.lock [-] Releasing lock
2021-05-18 20:12:57.146 68 ERROR armada.cli [-] Caught internal exception: armada.exceptions.armada_exceptions.ChartDeployException: Exception deploying charts: ['openstack-keystone']
2021-05-18 20:12:57.146 68 ERROR armada.cli Traceback (most recent call last):
2021-05-18 20:12:57.146 68 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/__init__.py", line 38, in safe_invoke
2021-05-18 20:12:57.146 68 ERROR armada.cli self.invoke()
2021-05-18 20:12:57.146 68 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 213, in invoke
2021-05-18 20:12:57.146 68 ERROR armada.cli resp = self.handle(documents, tiller)
2021-05-18 20:12:57.146 68 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py", line 81, in func_wrapper
2021-05-18 20:12:57.146 68 ERROR armada.cli return future.result()
2021-05-18 20:12:57.146 68 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
2021-05-18 20:12:57.146 68 ERROR armada.cli return self.__get_result()
2021-05-18 20:12:57.146 68 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2021-05-18 20:12:57.146 68 ERROR armada.cli raise self._exception
2021-05-18 20:12:57.146 68 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
2021-05-18 20:12:57.146 68 ERROR armada.cli result = self.fn(*self.args, **self.kwargs)
2021-05-18 20:12:57.146 68 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 256, in handle
2021-05-18 20:12:57.146 68 ERROR armada.cli return armada.sync()
2021-05-18 20:12:57.146 68 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 252, in sync
2021-05-18 20:12:57.146 68 ERROR armada.cli raise armada_exceptions.ChartDeployException(failures)
2021-05-18 20:12:57.146 68 ERROR armada.cli armada.exceptions.armada_exceptions.ChartDeployException: Exception deploying charts: ['openstack-keystone']
2021-05-18 20:12:57.146 68 ERROR armada.cli
command terminated with exit code 1

OpenStack Infra (hudson-openstack)
Changed in starlingx:
status: New → In Progress
Ghada Khalil (gkhalil)
tags: added: stx.containers stx.distro.openstack
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to openstack-armada-app (master)

Reviewed: https://review.opendev.org/c/starlingx/openstack-armada-app/+/792696
Committed: https://opendev.org/starlingx/openstack-armada-app/commit/4379649008955149209a9a80b9781c8a1baf4ede
Submitter: "Zuul (22348)"
Branch: master

commit 4379649008955149209a9a80b9781c8a1baf4ede
Author: Yvonne Ding <email address hidden>
Date: Fri May 21 15:54:25 2021 -0400

    Disallow application-apply when vim_progress_status is not enabled

    This fix is specific for AIO-SX because when node is unlocked/enabled/
    available the vim_progress_status could still be services-disabled.
    The status need a few more seconds to become services-enabled.

    Add a pre-check in openstack-armada-app/lifecycle_openstack.py to check
    AIO-SX node stable state before perform_app_apply. It prevents
    stx-openstack apply being triggered manually during initialization
    stage after node unlock.

    Closes-bug: 1929775
    Signed-off-by: Yvonne Ding <email address hidden>
    Change-Id: I563f77f617a68092b59f6cb38f5fb436a7933498

Changed in starlingx:
status: In Progress → Fix Released
Ghada Khalil (gkhalil)
Changed in starlingx:
importance: Undecided → Low
Revision history for this message
Ghada Khalil (gkhalil) wrote :

screening: marking as low / not gating - given the issue is reported as minor

Changed in starlingx:
assignee: nobody → Yvonne Ding (yding)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.