Brief Description
Custom application failed to apply on subcloud due to Kube Application timeout
Severity
Major
Steps to Reproduce
Deploy and manage 500 subclouds
Apply custom application on the subclouds, 250 per round
Expected Behavior
Custom application applied
Actual Behavior
Custom application failure due to Kube Application execution timeout.
Reproducibility
2 out of 500.
System Configuration
Distributed Cloud
Last Pass
2022-11-29
Timestamp/Logs
sysinv 2022-12-13 16:15:09.654 67535 ERROR sysinv.conductor.kube_app [-] Kube Application execution progress monitor timed out.: sysinv.common.exception.KubeAppProgressMonitorTimeout: Kube Application execution progress monitor timed out. 2022-12-13 16:15:09.654 67535 ERROR sysinv.conductor.kube_app Traceback (most recent call last): 2022-12-13 16:15:09.654 67535 ERROR sysinv.conductor.kube_app File "/usr/lib/python3/dist-packages/sysinv/conductor/kube_app.py", line 1892, in _make_fluxcd_operation_with_monitor 2022-12-13 16:15:09.654 67535 ERROR sysinv.conductor.kube_app rc = _check_progress() 2022-12-13 16:15:09.654 67535 ERROR sysinv.conductor.kube_app File "/usr/lib/python3/dist-packages/sysinv/conductor/kube_app.py", line 1870, in _check_progress 2022-12-13 16:15:09.654 67535 ERROR sysinv.conductor.kube_app time.sleep(1) 2022-12-13 16:15:09.654 67535 ERROR sysinv.conductor.kube_app File "/usr/lib/python3/dist-packages/eventlet/greenthread.py", line 36, in sleep 2022-12-13 16:15:09.654 67535 ERROR sysinv.conductor.kube_app hub.switch() 2022-12-13 16:15:09.654 67535 ERROR sysinv.conductor.kube_app File "/usr/lib/python3/dist-packages/eventlet/hubs/hub.py", line 298, in switch 2022-12-13 16:15:09.654 67535 ERROR sysinv.conductor.kube_app return self.greenlet.switch() 2022-12-13 16:15:09.654 67535 ERROR sysinv.conductor.kube_app sysinv.common.exception.KubeAppProgressMonitorTimeout: Kube Application execution progress monitor timed out. 2022-12-13 16:15:09.654 67535 ERROR sysinv.conductor.kube_app sysinv 2022-12-13 16:15:09.658 67535 INFO sysinv.conductor.kube_app [-] lifecycle hook for application analytics-app (22.12-0) started {'lifecycle_type': 'fluxcd-request', 'relative_timing': 'post', 'operation': 'apply', 'extra': {'rc': False}}. sysinv 2022-12-13 16:15:09.659 67535 INFO sysinv.conductor.kube_app [-] lifecycle hook for application analytics-app started {'mode': 'manual', 'lifecycle_type': 'rbd', 'relative_timing': 'post', 'operation': 'apply', 'extra': {}}. sysinv 2022-12-13 16:15:09.661 67535 INFO sysinv.conductor.kube_app [-] lifecycle hook for application analytics-app started {'mode': 'manual', 'lifecycle_type': 'resource', 'relative_timing': 'post', 'operation': 'apply', 'extra': {}}. sysinv 2022-12-13 16:15:09.905 67535 ERROR sysinv.conductor.kube_app [-] Application apply aborted!. sysinv 2022-12-13 16:15:09.905 67535 INFO sysinv.conductor.kube_app [-] Deregister the abort status of app analytics-app
Test Activity
Scalability Testing
Workaround
NA
Fix proposed to branch: master /review. opendev. org/c/starlingx /config/ +/869567
Review: https:/