Broken error message for stx-os apply rejection when services are not at services-enabled state

[Thales Elero Cervi]

Brief Description
-----------------
When a stx-openstack application apply operation is launched on a node in which VIM services are not at services-enabled state, the returned error message is broken and not providing useful information for the user.

Severity
--------
Minor: System/Feature is usable with minor issue

Steps to Reproduce
------------------
- Lock/unlock the controller
- Launch a manual "application-apply stx-openstack" right after the node reboot

Expected Behavior
------------------
The operation should be rejected, informing the reason to the user: "operation is not allowed while the node controller-0 not in services-enabled state."

Actual Behavior
----------------
The operation is rejected, with a broken error message: "Application-apply rejected: Remote error: %(exc_type)s %(value)s %(traceback)s."

Reproducibility
---------------
100% reproducible (given the node is not in services-enabled state)

System Configuration
--------------------
One node system (virtual environment)

Branch/Pull Time/Commit
-----------------------
master - 20220518

Last Pass
---------
Don't know.

Timestamp/Logs
--------------
[sysadmin@controller-0 ~(keystone_admin)]$ system application-apply stx-openstack
Application-apply rejected: Remote error: %(exc_type)s %(value)s
%(traceback)s.

from /var/log/sysinv.log:
sysinv 2022-05-19 13:40:37.218 98244 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672
sysinv 2022-05-19 13:40:37.225 98013 INFO sysinv.conductor.kube_app [-] lifecycle hook for application stx-openstack (1.0-199-centos-stable-versioned) started {'lifecycle_type': u'check', 'relative_timing': u'pre', 'mode': u'manual', 'operation': u'apply', 'extra': {}}.
sysinv 2022-05-19 13:40:37.251 98013 INFO k8sapp_openstack.lifecycle.lifecycle_openstack [-] stx-openstack requires application-apply, but some VIM services are not at services-enabled state on node controller-0.
Application-apply rejected.
sysinv 2022-05-19 13:40:37.252 98013 ERROR sysinv.openstack.common.rpc.amqp [-] Exception during message handling: LifecycleSemanticCheckException: Application-apply rejected: operation is not allowed while the node controller-0 not in services-enabled state.
2022-05-19 13:40:37.252 98013 ERROR sysinv.openstack.common.rpc.amqp Traceback (most recent call last):
2022-05-19 13:40:37.252 98013 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 436, in _process_data
2022-05-19 13:40:37.252 98013 ERROR sysinv.openstack.common.rpc.amqp **args)
2022-05-19 13:40:37.252 98013 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2022-05-19 13:40:37.252 98013 ERROR sysinv.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2022-05-19 13:40:37.252 98013 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 13795, in app_lifecycle_actions
2022-05-19 13:40:37.252 98013 ERROR sysinv.openstack.common.rpc.amqp self._app.app_lifecycle_actions(context, self, rpc_app, hook_info)
2022-05-19 13:40:37.252 98013 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 2379, in app_lifecycle_actions
2022-05-19 13:40:37.252 98013 ERROR sysinv.openstack.common.rpc.amqp lifecycle_op.app_lifecycle_actions(context, conductor_obj, self, app, hook_info)
2022-05-19 13:40:37.252 98013 ERROR sysinv.openstack.common.rpc.amqp File "/opt/platform/helm/22.06/stx-openstack/1.0-199-centos-stable-versioned/plugins/k8sapp_openstack/lifecycle/lifecycle_openstack.py", line
90, in app_lifecycle_actions
2022-05-19 13:40:37.252 98013 ERROR sysinv.openstack.common.rpc.amqp return self._pre_manual_apply_check(conductor_obj, app, hook_info)
2022-05-19 13:40:37.252 98013 ERROR sysinv.openstack.common.rpc.amqp File "/opt/platform/helm/22.06/stx-openstack/1.0-199-centos-stable-versioned/plugins/k8sapp_openstack/lifecycle/lifecycle_openstack.py", line
317, in _pre_manual_apply_check
2022-05-19 13:40:37.252 98013 ERROR sysinv.openstack.common.rpc.amqp active_controller.hostname, constants.VIM_SERVICES_ENABLED))
2022-05-19 13:40:37.252 98013 ERROR sysinv.openstack.common.rpc.amqp LifecycleSemanticCheckException: Application-apply rejected: operation is not allowed while the node controller-0 not in services-enabled state.
2022-05-19 13:40:37.252 98013 ERROR sysinv.openstack.common.rpc.amqp
sysinv 2022-05-19 13:40:37.260 98013 ERROR sysinv.openstack.common.rpc.common [-] Returning exception Application-apply rejected: operation is not allowed while the node controller-0 not in services-enabled state. to caller: LifecycleSemanticCheckException: Application-apply rejected: operation is not allowed while the node controller-0 not in services-enabled state.
sysinv 2022-05-19 13:40:37.260 98013 ERROR sysinv.openstack.common.rpc.common [-] ['Traceback (most recent call last):\n', ' File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 436, in _process_data\n **args)\n', ' File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch\n result = getattr(proxyobj, method)(ctxt, **kwargs)\n', ' File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 13795, in app_lifecycle_actions\n self._app.app_lifecycle_actions(context, self, rpc_app, hook_info)\n', ' File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 2379, in app_lifecycle_actions\n lifecycle_op.app_lifecycle_actions(context, conductor_obj, self, app, hook_info)\n', ' File "/opt/platform/helm/22.06/stx-openstack/1.0-199-centos-stable-versioned/plugins/k8sapp_openstack/lifecycle/lifecycle_openstack.py", line 90, in app_lifecycle_actions\n return self._pre_manual_apply_check(conductor_obj, app, hook_info)\n', ' File
"/opt/platform/helm/22.06/stx-openstack/1.0-199-centos-stable-versioned/plugins/k8sapp_openstack/lifecycle/lifecycle_openstack.py", line 317, in _pre_manual_apply_check\n active_controller.hostname, constants.VIM_SERVICES_ENABLED))\n', 'LifecycleSemanticCheckException: Application-apply rejected: operation is not allowed while the node controller-0 not in services-enabled state.\n']: LifecycleSemanticCheckException: Application-apply rejected: operation is not allowed while the node controller-0 not in services-enabled state.
sysinv 2022-05-19 13:40:37.272 98244 WARNING wsme.api [-] Client-side error: Application-apply rejected: Remote error: %(exc_type)s %(value)s
%(traceback)s.: ClientSideError: Application-apply rejected: Remote error: %(exc_type)s %(value)s

Test Activity
-------------
Developer Testing

Workaround
----------
None

[Ghada Khalil (gkhalil)]

@Thales, Is this an stx-openstack specific issue or a generic application framework issue?

[Thales Elero Cervi (tcervi)]

I just saw it when trying to apply the stx-openstack application on that system state, did not try with other apps.
Since the stx-openstack plugin appears to be generating the correct lifecyle message:

k8sapp_openstack.lifecycle.lifecycle_openstack [-] stx-openstack requires application-apply, but some VIM services are not at services-enabled state on node controller-0.
Application-apply rejected.

I would say that the wsme.api may have an issue when handling it:
wsme.api [-] Client-side error: Application-apply rejected: Remote error: %(exc_type)s %(value)s

But didn't have time to investigate it further so far.