The "openstack undercloud backup" got stuck for longer than an hour
openstack undercloud backup
(undercloud) [stack@undercloud ~]$ openstack undercloud backup
Waiting for messages on queue 'tripleo' with no timeout.
#####
/var/log/containers/mistral/engine.logXXX
-----------------------------------------
2019-01-22 07:10:51.410 6 INFO mistral.engine.engine_server [req-372e112c-b26b-40d3-b1cc-c36f4ff84491 14c61bf9e3734374b018c9ed35894e5e d90b0cfc52f84f3fb4eacec0b0549a87 - default default] Received RPC request 'on_action_complete'[action_ex_id=db833efc-cafa-4b63-b76c-8bba6aa01e1c, result=Result [data={created_at: 2019-01-21 16:55:54, description: Undercloud configuration parameters, variables: {u'undercloud_ceilometer_snmpd_password': u'QTmrfWCi3nVAjFIFI4Qj9leLS'}..., error=None, cancel=False]]
2019-01-22 07:10:51.424 6 INFO workflow_trace [req-372e112c-b26b-40d3-b1cc-c36f4ff84491 14c61bf9e3734374b018c9ed35894e5e d90b0cfc52f84f3fb4eacec0b0549a87 - default default] Action 'mistral.environments_get' (db833efc-cafa-4b63-b76c-8bba6aa01e1c)(task=get_database_credentials) [RUNNING -> SUCCESS, result = {created_at: 2019-01-21 16:55:54, description: Undercloud configuration parameters, variables: {u...]
2019-01-22 07:10:51.430 6 INFO workflow_trace [req-372e112c-b26b-40d3-b1cc-c36f4ff84491 14c61bf9e3734374b018c9ed35894e5e d90b0cfc52f84f3fb4eacec0b0549a87 - default default] Task 'get_database_credentials' (637efc92-f7cc-4420-88bb-897b13d3cf78) [RUNNING -> SUCCESS, msg=None] (execution_id=81583ade-bfbd-441f-9007-c7019014f4d5)
2019-01-22 07:10:51.431 6 DEBUG mistral.expressions.yaql_expression [req-372e112c-b26b-40d3-b1cc-c36f4ff84491 14c61bf9e3734374b018c9ed35894e5e d90b0cfc52f84f3fb4eacec0b0549a87 - default default] Start to evaluate YAQL expression. [expression='<% task().result.variables.undercloud_db_password %>', context={}] evaluate /usr/lib/python2.7/site-packages/mistral/expressions/yaql_expression.py:149
2019-01-22 07:10:51.441 6 ERROR mistral.engine.task_handler [req-372e112c-b26b-40d3-b1cc-c36f4ff84491 14c61bf9e3734374b018c9ed35894e5e d90b0cfc52f84f3fb4eacec0b0549a87 - default default] Failed to handle action completion [error=Can not evaluate YAQL expression [expression=task().result.variables.undercloud_db_password, error=u'undercloud_db_password', data={}], wf=tripleo.undercloud_backup.v1.backup, task=get_database_credentials, action=mistral.environments_get]:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/mistral/engine/task_handler.py", line 118, in _on_action_complete
task.on_action_complete(action_ex)
File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 160, in wrapper
result = f(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/mistral/engine/tasks.py", line 444, in on_action_complete
self.complete(state, state_info)
File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 160, in wrapper
result = f(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/mistral/engine/tasks.py", line 248, in complete
data_flow.publish_variables(self.task_ex, self.task_spec)
File "/usr/lib/python2.7/site-packages/mistral/workflow/data_flow.py", line 215, in publish_variables
task_ex.published = expr.evaluate_recursively(branch_vars, expr_ctx)
File "/usr/lib/python2.7/site-packages/mistral/expressions/__init__.py", line 100, in evaluate_recursively
data[key] = _evaluate_item(data[key], context)
File "/usr/lib/python2.7/site-packages/mistral/expressions/__init__.py", line 79, in _evaluate_item
return evaluate(item, context)
File "/usr/lib/python2.7/site-packages/mistral/expressions/__init__.py", line 71, in evaluate
return evaluator.evaluate(expression, context)
File "/usr/lib/python2.7/site-packages/mistral/expressions/yaql_expression.py", line 159, in evaluate
cls).evaluate(trim_expr, data_context)
File "/usr/lib/python2.7/site-packages/mistral/expressions/yaql_expression.py", line 113, in evaluate
", data=%s]" % (expression, str(e), data_context)
YaqlEvaluationException: Can not evaluate YAQL expression [expression=task().result.variables.undercloud_db_password, error=u'undercloud_db_password', data={}]
: YaqlEvaluationException: Can not evaluate YAQL expression [expression=task().result.variables.undercloud_db_password, error=u'undercloud_db_password', data={}]
####
https://github.com/openstack/tripleo-common/blob/master/workbooks/undercloud_backup.yaml#L47
# The Undercloud database password for the root
# user is stored in a Mistral environment, we
# need the password in order to run the database dump
get_database_credentials:
action: mistral.environments_get name='tripleo.undercloud-config'
publish:
status: SUCCESS
message: <% task().result %>
undercloud_db_password: <% task().result.variables.undercloud_db_password %>
on-success: create_database_backup
on-error: send_message
publish-on-error:
status: FAILED
message: <% task().result %>
#####
(undercloud) [stack@undercloud ~]$ mistral environment-get tripleo.undercloud-config
+-------------+-------------------------------------------------------------------------+
| Field | Value |
+-------------+-------------------------------------------------------------------------+
| Name | tripleo.undercloud-config |
| Description | Undercloud configuration parameters |
| Variables | { |
| | "undercloud_ceilometer_snmpd_password": "QTmrfWCi3nVAjFIFI4Qj9leLS" |
| | } |
| Scope | private |
| Created at | 2019-01-21 16:55:54 |
| Updated at | <none> |
+-------------+-------------------------------------------------------------------------+
Fix proposed to branch: master
Review: https:/