Task [Output for async deployment <role name>] fails with an undefined variable when async deployment times out

Bug #1958471 reported by Takashi Kajinami
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Medium
Takashi Kajinami

Bug Description

Description
===========
When the async task to run software deployment scripts times out, the subsequent task to dump stderr fails with the "The task includes an option with an undefined variable." message.
The output is cosmetic but it's better we avoid the error unrelated to the root cause.

~~~
2022-01-12 12:47:02,503 p=234567 u=mistral n=ansible | TASK [Run async deployment ControllerDeployment] *************************
...
2022-01-12 13:18:54,173 p=234567 u=mistral n=ansible | fatal: [controller-0]: FAILED! => {"changed": false, "msg": "async task did not complete within the requested time - 1800s"}
2022-01-12 13:18:54,173 p=234567 u=mistral n=ansible | ...ignoring
...
2022-01-12 13:18:58,959 p=234567 u=mistral n=ansible | TASK [Output for async deployment ControllerDeployment] ******************
...
2022-01-12 13:18:58,996 p=234567 u=mistral n=ansible | fatal: [controller-0]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'stderr'\n\nThe error appears to be in '/var/lib/mistral/overcloud/deployments.yaml': line 193, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: \"Output for async deployment {{ item }}\"\n ^ here\nWe could be wrong, but this one looks like it might be an issue with\nmissing quotes. Always quote template expression brackets when they\nstart a value. For instance:\n\n with_items:\n - {{ foo }}\n\nShould be written as:\n\n with_items:\n - \"{{ foo }}\"\n"}
...
2022-01-12 13:18:59,050 p=234567 u=mistral n=ansible | NO MORE HOSTS LEFT *************************************************************
2022-01-12 13:18:59,051 p=234567 u=mistral n=ansible | PLAY RECAP *********************************************************************
...
2022-01-12 13:18:59,052 p=234567 u=mistral n=ansible | controller-0 : ok=67 changed=35 unreachable=0 failed=1 skipped=43 rescued=0 ignored=1
...
~~~

Steps to reproduce
==================
1. Implement a long sleep task (>1800 seconds) for ExtraConfigPre resource
2. Run deployment

Expected result
===============
Deployment fails and the undefined variable error is not shown

Actual result
=============
Deployment fails and the undefined variable error is shown

Environment
===========
This issue was initially found in stable/train based deloyment
and reported in https://bugzilla.redhat.com/show_bug.cgi?id=2041493 .

Logs & Configs
==============
N/A

Changed in tripleo:
importance: Undecided → Low
description: updated
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (master)
Changed in tripleo:
status: New → In Progress
Changed in tripleo:
importance: Low → Medium
tags: added: ux
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (master)

Reviewed: https://review.opendev.org/c/openstack/tripleo-common/+/825418
Committed: https://opendev.org/openstack/tripleo-common/commit/2cd1fbd87bfde7ec318356178d1ca4c68859e916
Submitter: "Zuul (22348)"
Branch: master

commit 2cd1fbd87bfde7ec318356178d1ca4c68859e916
Author: Takashi Kajinami <email address hidden>
Date: Thu Jan 20 10:42:04 2022 +0900

    Fix reference to undefined variables when heat-config times out

    When an async task times out, it doesn't register its result and
    the subsequent task which refers to the result can't look up details
    like stderr output or return code.

    This change defines the default values to fix reference to unefined
    variables.

    Closes-Bug: #1958471
    Change-Id: I5049e88872bfe3aa8072804cb67e8f7178951d9d

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-common/+/825743

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/tripleo-common/+/825743
Committed: https://opendev.org/openstack/tripleo-common/commit/64e0dc7fdf69bb4363764efcf4536f938803c76e
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 64e0dc7fdf69bb4363764efcf4536f938803c76e
Author: Takashi Kajinami <email address hidden>
Date: Thu Jan 20 10:42:04 2022 +0900

    Fix reference to undefined variables when heat-config times out

    When an async task times out, it doesn't register its result and
    the subsequent task which refers to the result can't look up details
    like stderr output or return code.

    This change defines the default values to fix reference to unefined
    variables.

    Closes-Bug: #1958471
    Change-Id: I5049e88872bfe3aa8072804cb67e8f7178951d9d
    (cherry picked from commit 2cd1fbd87bfde7ec318356178d1ca4c68859e916)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/tripleo-common/+/828848

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/tripleo-common/+/828849

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/tripleo-common/+/829030

Changed in tripleo:
assignee: nobody → Takashi Kajinami (kajinamit)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/tripleo-common/+/828848
Committed: https://opendev.org/openstack/tripleo-common/commit/f105287f17242ef7570a34b6817af3032edc912a
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit f105287f17242ef7570a34b6817af3032edc912a
Author: Takashi Kajinami <email address hidden>
Date: Thu Jan 20 10:42:04 2022 +0900

    Fix reference to undefined variables when heat-config times out

    When an async task times out, it doesn't register its result and
    the subsequent task which refers to the result can't look up details
    like stderr output or return code.

    This change defines the default values to fix reference to unefined
    variables.

    Closes-Bug: #1958471
    Change-Id: I5049e88872bfe3aa8072804cb67e8f7178951d9d
    (cherry picked from commit 2cd1fbd87bfde7ec318356178d1ca4c68859e916)
    (cherry picked from commit 64e0dc7fdf69bb4363764efcf4536f938803c76e)

tags: added: in-stable-victoria
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-common (stable/ussuri)

Change abandoned by "Marios Andreou <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/tripleo-common/+/828849
Reason: abandoning per http://lists.openstack.org/pipermail/openstack-discuss/2022-April/028026.html - so we can move EOL https://review.opendev.org/c/openstack/releases/+/834049

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-common 16.4.0

This issue was fixed in the openstack/tripleo-common 16.4.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-common 13.3.0

This issue was fixed in the openstack/tripleo-common 13.3.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (stable/train)

Reviewed: https://review.opendev.org/c/openstack/tripleo-common/+/829030
Committed: https://opendev.org/openstack/tripleo-common/commit/f8e854a6737dee0663d6e6e732e209a317a9dfa0
Submitter: "Zuul (22348)"
Branch: stable/train

commit f8e854a6737dee0663d6e6e732e209a317a9dfa0
Author: Takashi Kajinami <email address hidden>
Date: Thu Jan 20 10:42:04 2022 +0900

    Fix reference to undefined variables when heat-config times out

    When an async task times out, it doesn't register its result and
    the subsequent task which refers to the result can't look up details
    like stderr output or return code.

    This change defines the default values to fix reference to unefined
    variables.

    Closes-Bug: #1958471
    Change-Id: I5049e88872bfe3aa8072804cb67e8f7178951d9d
    (cherry picked from commit 2cd1fbd87bfde7ec318356178d1ca4c68859e916)
    (cherry picked from commit 64e0dc7fdf69bb4363764efcf4536f938803c76e)
    (cherry picked from commit f105287f17242ef7570a34b6817af3032edc912a)

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-common train-eol

This issue was fixed in the openstack/tripleo-common train-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.