master deployment fails on Task 'check_flavors' error=Can not evaluate YAQL expression

Bug #1723171 reported by wes hayutin on 2017-10-12
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Critical
Adriano Petrich

Bug Description

https://logs.rdoproject.org/openstack-periodic-4hr/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/418a7ec/undercloud/var/log/mistral/executor.log.txt.gz#_2017-10-12_12_58_44_524

2017-10-12 12:58:44.524 32445 ERROR mistral.executors.default_executor [req-d44299af-8e24-4387-926b-f3de65425620 1eb002dec602472f93300b97c1c0bbf8 95e403e8d1ed42d0ab497913861f926d - default default] Failed to run action [action_ex_id=3503c349-af40-48fd-9a94-deb7a82add78, action_cls='<class 'mistral.actions.action_factory.CheckFlavorsAction'>', attributes='{}', params='{u'roles_info': {u'control': [None, None], u'compute': [None, None], u'block-storage': [None, None], u'ceph-storage': [None, None], u'swift-storage': [None, None]}}']
 (https://192.168.24.2:13000/v2.0/tokens): The resource could not be found. (HTTP 404) (Request-ID: req-bd7652b8-d537-4f3e-b191-7cf23ef1d17a): NotFound: (https://192.168.24.2:13000/v2.0/tokens): The resource could not be found. (HTTP 404) (Request-ID: req-bd7652b8-d537-4f3e-b191-7cf23ef1d17a)

===============================================================================
2017-10-12 12:58:44.659 32378 ERROR mistral.engine.task_handler [req-dbdfa17b-a7ef-46c5-86b5-20f35e9df035 1eb002dec602472f93300b97c1c0bbf8 95e403e8d1ed42d0ab497913861f926d - default default] Failed to handle action completion [error=Can not evaluate YAQL expression [expression=task(check_flavors).result.errors, error=Unknown function "#property#errors", data={}], wf=tripleo.validations.v1.collect_flavors, task=check_flavors, action=tripleo.validations.check_flavors]:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/mistral/engine/task_handler.py", line 110, in _on_action_complete
    task.on_action_complete(action_ex)
  File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 157, in wrapper
    result = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/mistral/engine/tasks.py", line 306, in on_action_complete
    self.complete(state, state_info)
  File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 157, in wrapper
    result = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/mistral/engine/tasks.py", line 172, in complete
    data_flow.publish_variables(self.task_ex, self.task_spec)
  File "/usr/lib/python2.7/site-packages/mistral/workflow/data_flow.py", line 210, in publish_variables
    task_ex.published = expr.evaluate_recursively(branch_vars, expr_ctx)
  File "/usr/lib/python2.7/site-packages/mistral/expressions/__init__.py", line 100, in evaluate_recursively
    data[key] = _evaluate_item(data[key], context)
  File "/usr/lib/python2.7/site-packages/mistral/expressions/__init__.py", line 79, in _evaluate_item
    return evaluate(item, context)
  File "/usr/lib/python2.7/site-packages/mistral/expressions/__init__.py", line 71, in evaluate
    return evaluator.evaluate(expression, context)
  File "/usr/lib/python2.7/site-packages/mistral/expressions/yaql_expression.py", line 100, in evaluate
    cls).evaluate(trim_expr, data_context)
  File "/usr/lib/python2.7/site-packages/mistral/expressions/yaql_expression.py", line 54, in evaluate
    ", data=%s]" % (expression, str(e), data_context)
YaqlEvaluationException: Can not evaluate YAQL expression [expression=task(check_flavors).result.errors, error=Unknown function "#property#errors", data={}]
: YaqlEvaluationException: Can not evaluate YAQL expression [expression=task(check_flavors).result.errors, error=Unknown function "#property#errors", data={}]
2017-10-12 12:58:44.659 32378 INFO workflow_trace [req-dbdfa17b-a7ef-46c5-86b5-20f35e9df035 1eb002dec602472f93300b97c1c0bbf8 95e403e8d1ed42d0ab497913861f926d - default default] Task 'check_flavors' (be11e568-7ab2-4b1a-a1e0-9f9fb8ae789c) [ERROR -> ERROR, msg=Failed to handle action completion [error=Can not evaluate YAQL expression [expression=task(check_flavors).result.errors, error=Unknown function "#property#errors", data={}], wf=tripleo.validations.v1.collect_flavors, task=check_flavors, action=tripleo.validations.check_flavors]:

https://logs.rdoproject.org/openstack-periodic-4hr/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/418a7ec/undercloud/var/log/mistral/engine.log.txt.gz#_2017-10-12_12_58_44_659

=====================================================================

| 8cb39f4e-7815-46dc-9893-26e656ba846d | 2ce3aceb-5215-41a9-8c6c-d34f7ba6a4a7 | tripleo.validations.v1.check_pre_deployment_validations | | <none> | ERROR | Failed to handle action c... | 2017-10-12 12:58:41 | 2017-10-12 12:58:44 |
| 562dcabb-b51f-495e-ac03-001301602c52 | d2cd8db0-92c7-465e-95f1-308cba1ff8d4 | tripleo.validations.v1.collect_flavors | sub-workflow execution | 2d0e462f-a237-46a6-8e8c-2c376dae3711 | ERROR | Failed to handle action c... | 2017-10-12 12:58:44 | 2017-10-12 12:58:44 |
| 85766095-e12e-4b78-9075-78df578b2a3d | 81889b8d-0d6d-4c9a-b6a2-85f89b274006 | tripleo.plan_management.v1.publish_ui_logs_to_swift | {"triggered_by": {"type": "cron_trigger", "id": "5ce16f15-cb07-4593-a199-7e88fe352166", "name": "publish-ui-logs-hourly"}, "description": "Workflow execution created by cron trigger '(5ce16f15-cb07-4593-a199-7e88fe352166)'."} | <none> | SUCCESS | None | 2017-10-12 12:59:59 | 2017-10-12 13:00:06 |

https://logs.rdoproject.org/openstack-periodic-4hr/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/418a7ec/postci.txt.gz

========================================================================

Flavors listed here:
https://logs.rdoproject.org/openstack-periodic-4hr/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/418a7ec/undercloud/home/jenkins/undercloud_install.log.txt.gz#_2017-10-12_12_33_17

wes hayutin (weshayutin) on 2017-10-12
tags: added: alert promotion-blocker
removed: prom
Dougal Matthews (d0ugal) on 2017-10-12
tags: added: workflows
Changed in tripleo:
assignee: nobody → Adriano Petrich (apetrich)
Adriano Petrich (apetrich) wrote :

The error with "Can not evaluate YAQL expression [expression=task(check_flavors).result.errors, error=Unknown function "#property#errors", data={}]" is a red herring it is an error that we need to fix but it is happening because we are passing

u'roles_info': {u'control': [None, None], u'compute': [None, None], u'block-storage': [None, None], u'ceph-storage': [None, None], u'swift-storage': [None, None]}}'

That comes from here https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/overcloud_deploy.py#L701
that comes from this util function
https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/utils.py#L604,L618

From what I can see on the overcloud-deploy.sh we are not passing "--control-flavor oooq_control --compute-flavor oooq_compute --ceph-storage-flavor oooq_ceph --block-storage-flavor oooq_blockstorage" that we used to pass

here:

https://logs.rdoproject.org/openstack-periodic-4hr/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/418a7ec/undercloud/home/jenkins/overcloud_deploy.log.txt.gz#_2017-10-12_12_58_37

Adriano Petrich (apetrich) wrote :

<ykarel> bought to my attention that it might be an issue with novaclient using keystone v2 parameters specially here

https://github.com/openstack/tripleo-common/blob/master/tripleo_common/actions/base.py#L123-L132

tripleo.utils.keystone.py has a fix for keystone we just need to fix either novaclient or the call to it on those lines

Dougal Matthews (d0ugal) wrote :

I opened another bug for the red-herring error. The YAQL error is actually the error handling failing. https://bugs.launchpad.net/tripleo/+bug/1723170

yatin (yatinkarel) wrote :

> 2 there is nova api v2, not keystone v2
although, best would be to use keystoneauth session https://docs.openstack.org/python-novaclient/pike/reference/api/index.html

As project_domain_name and user_domain_name were not passed in creating novaclient at https://github.com/openstack/tripleo-common/blob/master/tripleo_common/actions/base.py#L123-L132, it tries to reach keystone v2(non existing) and fails. and Yes using keystoneauth session would be better.
Info: I tried passing project_domain_name and user_domain_name while creating novaclient and tripleo validations(failing here) and it passed.

Fix proposed to branch: master
Review: https://review.openstack.org/512219

Changed in tripleo:
status: Triaged → In Progress

Reviewed: https://review.openstack.org/512219
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=e9a82e329ee03ba02ccc6776db651a7fa1e20037
Submitter: Zuul
Branch: master

commit e9a82e329ee03ba02ccc6776db651a7fa1e20037
Author: Adriano Petrich <email address hidden>
Date: Mon Oct 16 09:49:54 2017 +0100

    Add user and project_user_domains to novaclient

    As project_domain_name and user_domain_name were not passed in creating
    novaclient it tries to reach keystone v2(non existing) and fails.

    Change-Id: I9a952f8c8eef5d9fc1d331b6c2da26084526ad8d
    Closes-Bug: #1723171

Changed in tripleo:
status: In Progress → Fix Released

This issue was fixed in the openstack/tripleo-common 8.1.0 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers