master deployment fails on Task 'check_flavors' error=Can not evaluate YAQL expression

Bug #1723171 reported by wes hayutin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Adriano Petrich

Bug Description

https://logs.rdoproject.org/openstack-periodic-4hr/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/418a7ec/undercloud/var/log/mistral/executor.log.txt.gz#_2017-10-12_12_58_44_524

2017-10-12 12:58:44.524 32445 ERROR mistral.executors.default_executor [req-d44299af-8e24-4387-926b-f3de65425620 1eb002dec602472f93300b97c1c0bbf8 95e403e8d1ed42d0ab497913861f926d - default default] Failed to run action [action_ex_id=3503c349-af40-48fd-9a94-deb7a82add78, action_cls='<class 'mistral.actions.action_factory.CheckFlavorsAction'>', attributes='{}', params='{u'roles_info': {u'control': [None, None], u'compute': [None, None], u'block-storage': [None, None], u'ceph-storage': [None, None], u'swift-storage': [None, None]}}']
 (https://192.168.24.2:13000/v2.0/tokens): The resource could not be found. (HTTP 404) (Request-ID: req-bd7652b8-d537-4f3e-b191-7cf23ef1d17a): NotFound: (https://192.168.24.2:13000/v2.0/tokens): The resource could not be found. (HTTP 404) (Request-ID: req-bd7652b8-d537-4f3e-b191-7cf23ef1d17a)

===============================================================================
2017-10-12 12:58:44.659 32378 ERROR mistral.engine.task_handler [req-dbdfa17b-a7ef-46c5-86b5-20f35e9df035 1eb002dec602472f93300b97c1c0bbf8 95e403e8d1ed42d0ab497913861f926d - default default] Failed to handle action completion [error=Can not evaluate YAQL expression [expression=task(check_flavors).result.errors, error=Unknown function "#property#errors", data={}], wf=tripleo.validations.v1.collect_flavors, task=check_flavors, action=tripleo.validations.check_flavors]:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/mistral/engine/task_handler.py", line 110, in _on_action_complete
    task.on_action_complete(action_ex)
  File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 157, in wrapper
    result = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/mistral/engine/tasks.py", line 306, in on_action_complete
    self.complete(state, state_info)
  File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 157, in wrapper
    result = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/mistral/engine/tasks.py", line 172, in complete
    data_flow.publish_variables(self.task_ex, self.task_spec)
  File "/usr/lib/python2.7/site-packages/mistral/workflow/data_flow.py", line 210, in publish_variables
    task_ex.published = expr.evaluate_recursively(branch_vars, expr_ctx)
  File "/usr/lib/python2.7/site-packages/mistral/expressions/__init__.py", line 100, in evaluate_recursively
    data[key] = _evaluate_item(data[key], context)
  File "/usr/lib/python2.7/site-packages/mistral/expressions/__init__.py", line 79, in _evaluate_item
    return evaluate(item, context)
  File "/usr/lib/python2.7/site-packages/mistral/expressions/__init__.py", line 71, in evaluate
    return evaluator.evaluate(expression, context)
  File "/usr/lib/python2.7/site-packages/mistral/expressions/yaql_expression.py", line 100, in evaluate
    cls).evaluate(trim_expr, data_context)
  File "/usr/lib/python2.7/site-packages/mistral/expressions/yaql_expression.py", line 54, in evaluate
    ", data=%s]" % (expression, str(e), data_context)
YaqlEvaluationException: Can not evaluate YAQL expression [expression=task(check_flavors).result.errors, error=Unknown function "#property#errors", data={}]
: YaqlEvaluationException: Can not evaluate YAQL expression [expression=task(check_flavors).result.errors, error=Unknown function "#property#errors", data={}]
2017-10-12 12:58:44.659 32378 INFO workflow_trace [req-dbdfa17b-a7ef-46c5-86b5-20f35e9df035 1eb002dec602472f93300b97c1c0bbf8 95e403e8d1ed42d0ab497913861f926d - default default] Task 'check_flavors' (be11e568-7ab2-4b1a-a1e0-9f9fb8ae789c) [ERROR -> ERROR, msg=Failed to handle action completion [error=Can not evaluate YAQL expression [expression=task(check_flavors).result.errors, error=Unknown function "#property#errors", data={}], wf=tripleo.validations.v1.collect_flavors, task=check_flavors, action=tripleo.validations.check_flavors]:

https://logs.rdoproject.org/openstack-periodic-4hr/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/418a7ec/undercloud/var/log/mistral/engine.log.txt.gz#_2017-10-12_12_58_44_659

=====================================================================

| 8cb39f4e-7815-46dc-9893-26e656ba846d | 2ce3aceb-5215-41a9-8c6c-d34f7ba6a4a7 | tripleo.validations.v1.check_pre_deployment_validations | | <none> | ERROR | Failed to handle action c... | 2017-10-12 12:58:41 | 2017-10-12 12:58:44 |
| 562dcabb-b51f-495e-ac03-001301602c52 | d2cd8db0-92c7-465e-95f1-308cba1ff8d4 | tripleo.validations.v1.collect_flavors | sub-workflow execution | 2d0e462f-a237-46a6-8e8c-2c376dae3711 | ERROR | Failed to handle action c... | 2017-10-12 12:58:44 | 2017-10-12 12:58:44 |
| 85766095-e12e-4b78-9075-78df578b2a3d | 81889b8d-0d6d-4c9a-b6a2-85f89b274006 | tripleo.plan_management.v1.publish_ui_logs_to_swift | {"triggered_by": {"type": "cron_trigger", "id": "5ce16f15-cb07-4593-a199-7e88fe352166", "name": "publish-ui-logs-hourly"}, "description": "Workflow execution created by cron trigger '(5ce16f15-cb07-4593-a199-7e88fe352166)'."} | <none> | SUCCESS | None | 2017-10-12 12:59:59 | 2017-10-12 13:00:06 |

https://logs.rdoproject.org/openstack-periodic-4hr/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/418a7ec/postci.txt.gz

========================================================================

Flavors listed here:
https://logs.rdoproject.org/openstack-periodic-4hr/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/418a7ec/undercloud/home/jenkins/undercloud_install.log.txt.gz#_2017-10-12_12_33_17

wes hayutin (weshayutin)
tags: added: alert promotion-blocker
removed: prom
Dougal Matthews (d0ugal)
tags: added: workflows
Changed in tripleo:
assignee: nobody → Adriano Petrich (apetrich)
Revision history for this message
Adriano Petrich (apetrich) wrote :

The error with "Can not evaluate YAQL expression [expression=task(check_flavors).result.errors, error=Unknown function "#property#errors", data={}]" is a red herring it is an error that we need to fix but it is happening because we are passing

u'roles_info': {u'control': [None, None], u'compute': [None, None], u'block-storage': [None, None], u'ceph-storage': [None, None], u'swift-storage': [None, None]}}'

That comes from here https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/overcloud_deploy.py#L701
that comes from this util function
https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/utils.py#L604,L618

From what I can see on the overcloud-deploy.sh we are not passing "--control-flavor oooq_control --compute-flavor oooq_compute --ceph-storage-flavor oooq_ceph --block-storage-flavor oooq_blockstorage" that we used to pass

here:

https://logs.rdoproject.org/openstack-periodic-4hr/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/418a7ec/undercloud/home/jenkins/overcloud_deploy.log.txt.gz#_2017-10-12_12_58_37

Revision history for this message
Adriano Petrich (apetrich) wrote :

<ykarel> bought to my attention that it might be an issue with novaclient using keystone v2 parameters specially here

https://github.com/openstack/tripleo-common/blob/master/tripleo_common/actions/base.py#L123-L132

tripleo.utils.keystone.py has a fix for keystone we just need to fix either novaclient or the call to it on those lines

Revision history for this message
Dougal Matthews (d0ugal) wrote :

I opened another bug for the red-herring error. The YAQL error is actually the error handling failing. https://bugs.launchpad.net/tripleo/+bug/1723170

Revision history for this message
Alan Pevec (apevec) wrote :
Revision history for this message
yatin (yatinkarel) wrote :

> 2 there is nova api v2, not keystone v2
although, best would be to use keystoneauth session https://docs.openstack.org/python-novaclient/pike/reference/api/index.html

As project_domain_name and user_domain_name were not passed in creating novaclient at https://github.com/openstack/tripleo-common/blob/master/tripleo_common/actions/base.py#L123-L132, it tries to reach keystone v2(non existing) and fails. and Yes using keystoneauth session would be better.
Info: I tried passing project_domain_name and user_domain_name while creating novaclient and tripleo validations(failing here) and it passed.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (master)

Fix proposed to branch: master
Review: https://review.openstack.org/512219

Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (master)

Reviewed: https://review.openstack.org/512219
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=e9a82e329ee03ba02ccc6776db651a7fa1e20037
Submitter: Zuul
Branch: master

commit e9a82e329ee03ba02ccc6776db651a7fa1e20037
Author: Adriano Petrich <email address hidden>
Date: Mon Oct 16 09:49:54 2017 +0100

    Add user and project_user_domains to novaclient

    As project_domain_name and user_domain_name were not passed in creating
    novaclient it tries to reach keystone v2(non existing) and fails.

    Change-Id: I9a952f8c8eef5d9fc1d331b6c2da26084526ad8d
    Closes-Bug: #1723171

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-common 8.1.0

This issue was fixed in the openstack/tripleo-common 8.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.