multinode promotion jobs timing out at overcloud deploy

[Quique Llorente]

http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset018-master/842c47d/job-output.txt.gz
2019-02-15 00:58:13.283460 | primary | Friday 15 February 2019 00:58:13 +0000 (0:00:00.068) 0:00:31.024 *******
2019-02-15 00:58:13.307743 | primary | skipping: [undercloud]
2019-02-15 00:58:13.331631 | primary |
2019-02-15 00:58:13.331818 | primary | TASK [overcloud-deploy : Deploy the overcloud] *********************************
2019-02-15 00:58:13.353011 | primary | Friday 15 February 2019 00:58:13 +0000 (0:00:00.069) 0:00:31.093 *******
2019-02-15 03:19:01.213841 | RUN END RESULT_TIMED_OUT: [untrusted : git.openstack.org/openstack-infra/tripleo-ci/playbooks/tripleo-ci/run-v3.yaml@master]
2019-02-15 03:19:01.214355 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/tripleo-ci-periodic-base/post.yaml@master]
2019-02-15 03:19:03.457998 |
2019-02-15 03:19:03.458239 | PLAY [primary:tripleo-ovb-centos-7]
2019-02-15 03:19:03.575181 |
2019-02-15 03:19:03.575434 | TASK [Set zuul-log-path fact]
2019-02-15 03:19:03.641573 | primary | ok
2019-02-15 03:19:03.696398 |
2019-02-15 03:19:03.696643 | TASK [set-zuul-log-path-fact : Set log path for a change]
2019-02-15 03:19:03.778682 | primary | skipping: Conditional result was False
2019-02-15 03:19:03.818512 |
2019-02-15 03:19:03.818883 | TASK [set-zuul-log-path-fact : Set log path for a ref update]
2019-02-15 03:19:03.894968 | primary | skipping: Conditional result was False
2

[chandan kumar (chkumar246)]

fs37, 30, 19, 18, 17, 16 in all of them got timeout.

[chandan kumar (chkumar246)]

From mistral engine.log, it appears the following error caused the timedout
https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-master/622f9f3/logs/undercloud/var/log/containers/mistral/engine.log.txt.gz?level=ERROR

2019-02-15 00:48:11.534 8 ERROR mistral.engine.task_handler [req-5d25ed5f-19f3-427a-a3b1-6e732b19231a ab1aa24494f14abfa681a06c05a83e76 0561c56b74b84014972d46763e42034f - default default] Failed to run task [error=Can not evaluate YAQL expression [expression=$.running_config_download_workflows[0].id, error=list index out of range, data={}], wf=tripleo.deployment.v1.config_download_deploy, task=fail_deploy_in_progress]:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/mistral/engine/task_handler.py", line 63, in run_task
    task.run()
  File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 160, in wrapper
    result = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/mistral/engine/tasks.py", line 453, in run
    self._run_new()
  File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 160, in wrapper
    result = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/mistral/engine/tasks.py", line 485, in _run_new
    self._schedule_actions()
  File "/usr/lib/python2.7/site-packages/mistral/engine/tasks.py", line 558, in _schedule_actions
    input_dict = self._get_action_input()
  File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 160, in wrapper
    result = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/mistral/engine/tasks.py", line 589, in _get_action_input
    input_dict = self._evaluate_expression(self.task_spec.get_input(), ctx)
  File "/usr/lib/python2.7/site-packages/mistral/engine/tasks.py", line 615, in _evaluate_expression
    ctx_view
  File "/usr/lib/python2.7/site-packages/mistral/expressions/__init__.py", line 100, in evaluate_recursively
    data[key] = _evaluate_item(data[key], context)
  File "/usr/lib/python2.7/site-packages/mistral/expressions/__init__.py", line 79, in _evaluate_item
    return evaluate(item, context)
  File "/usr/lib/python2.7/site-packages/mistral/expressions/__init__.py", line 71, in evaluate
    return evaluator.evaluate(expression, context)
  File "/usr/lib/python2.7/site-packages/mistral/expressions/yaql_expression.py", line 159, in evaluate
    cls).evaluate(trim_expr, data_context)
  File "/usr/lib/python2.7/site-packages/mistral/expressions/yaql_expression.py", line 113, in evaluate
    ", data=%s]" % (expression, str(e), data_context)
YaqlEvaluationException: Can not evaluate YAQL expression [expression=$.running_config_download_workflows[0].id, error=list index out of range, data={}]
: YaqlEvaluationException: Can not evaluate YAQL expression [expression=$.running_config_download_workflows[0].id, error=list index out of range, data={}]

[chandan kumar (chkumar246)]

It also affects fs010.

[Quique Llorente (quiquell)]

Feels related to
https://github.com/openstack/mistral/commit/a39db2d3dc21afbe8b9e1d6f8dd870da272b9671

[chandan kumar (chkumar246)]

Reverted the change: https://review.openstack.org/#/c/637170/ and testing here: https://review.rdoproject.org/r/#/c/13943/

[chandan kumar (chkumar246)]

Based on recent run against test-project in both cases it is passing.
https://logs.rdoproject.org/43/13943/82/check/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-master/4b53345/job-output.txt.gz#_2019-02-15_14_22_17_063214

and

https://logs.rdoproject.org/49/18749/3/check/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-master/5873fce/job-output.txt.gz#_2019-02-15_14_49_44_071880

[chandan kumar (chkumar246)]

https://github.com/openstack/tripleo-common/blob/master/workbooks/deployment.yaml#L404 -> returning true somewhow

[chandan kumar (chkumar246)]

From check job:
https://logs.rdoproject.org/49/18749/3/check/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-master/5873fce/logs/undercloud/var/log/extra/podman/containers/mistral_api/podman_info.log.txt.gz

openstack-mistral-api.noarch 8.0.0-0.20190211115159.58d6634.el7 @delorean-> https://github.com/openstack/mistral/commit/58d6634702014eb3c2bf70dbe487fe5983cef8e9

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-master/7e19d72/logs/undercloud/var/log/extra/podman/containers/mistral_api/podman_info.log.txt.gz

openstack-mistral-api.noarch 8.0.0-0.20190214191609.a39db2d.el7 @delorean -> https://github.com/openstack/mistral/commit/a39db2d3dc21afbe8b9e1d6f8dd870da272b9671

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/637488

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/637522

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/637546

[OpenStack Infra (hudson-openstack)]

Change abandoned by Chandan Kumar (<email address hidden>) on branch: master
Review: https://review.openstack.org/637488
Reason: in favor of https://review.openstack.org/#/c/637546/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/637522
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=573f67df3d6216f62467eb526a15be3120f9a01a
Submitter: Zuul
Branch: master

commit 573f67df3d6216f62467eb526a15be3120f9a01a
Author: Quique Llorente <email address hidden>
Date: Mon Feb 18 11:53:19 2019 +0100

    Use len to compare list at mistral

    Looks like mistral has issues using empty list [] comparation to check
    if the list is empty or not, let's just use len.

    This is needed since https://review.openstack.org/#/c/637507/ is not
    merged.

    Change-Id: Ifd04d741bd92b2ada5f181ed87ed5b07c0e01d48
    Closes-Bug: #1816026

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/637546
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=d1fd33e82f6c3eb4750561c6f50122d1026bad08
Submitter: Zuul
Branch: master

commit d1fd33e82f6c3eb4750561c6f50122d1026bad08
Author: Chandan Kumar <email address hidden>
Date: Mon Feb 18 19:28:49 2019 +0530

    Set container_binary to podman from stein releases onwards

    https://review.openstack.org/#/c/608452/ Switch the undercloud to
    deploy Podman by default starting from Stein and
    undercloud_container_cli var is setted under undercloud_deploy role
    and it cannot be used in another role which will set
    container_binary to docker but docker is not there in stein leading
    to fail running tempest from container.

    Use sudo while pulling tempest using podman

    Change-Id: I7ee75d71a7c34bdb85232a1378880ff16579c5db
    Related-Bug: #1816026

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 10.5.0 release.