Existing requirement_file and constraint_file may result in failed playbook runs following failure

Bug #1824189 reported by James Denton on 2019-04-10
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
openstack-ansible
Status tracked in Train
Stein
Medium
Mohammed Naser
Train
Medium
Mohammed Naser

Bug Description

OS: Ubuntu 18.04

DISTRIB_ID="OSA"
DISTRIB_RELEASE="19.1.0.dev14"
DISTRIB_CODENAME="Stein"
DISTRIB_DESCRIPTION="OpenStack-Ansible"

While deploying OSA, I found that if a playbook run fails or is cancelled prematurely, subsequent runs may also fail due to certain tasks being skipped based on 'when' conditions noted in this bug.

In this example, the 'Install python packages into the venv' task failed during the first run due to a lack of disk space:

TASK [python_venv_build : Install python packages into the venv] *********************************************************************
FAILED - RETRYING: Install python packages into the venv (5 retries left).
changed: [denver-infra01_ceilometer_central_container-9766bf53]
FAILED - RETRYING: Install python packages into the venv (4 retries left).
FAILED - RETRYING: Install python packages into the venv (3 retries left).
FAILED - RETRYING: Install python packages into the venv (2 retries left).
FAILED - RETRYING: Install python packages into the venv (1 retries left).
fatal: [denver-compute01]: FAILED! =>
...
Could not install packages due to an EnvironmentError: [Errno 28] No space left on device

After enlarging the disk, on a subsequent run, the task was skipped and the failure occurred here:

TASK [python_venv_build : Install python packages into the venv] *********************************************************************
skipping: [denver-infra01_ceilometer_central_container-9766bf53]
skipping: [denver-compute01]

...

TASK [os_ceilometer : Preserve original configuration file(s)] ***********************************************************************
 [WARNING]: when statements should not include jinja2 templating delimiters such as {{ }} or {% %}. Found: {{ item.condition |
default(True) }}

 [WARNING]: when statements should not include jinja2 templating delimiters such as {{ }} or {% %}. Found: {{ item.condition |
default(True) }}

ok: [denver-infra01_ceilometer_central_container-9766bf53] => (item={u'source_f': u'/etc/ceilometer/rootwrap.conf', u'tmp_f': u'/tmp/rootwrap.conf', u'config_type': u'ini', u'group': u'ceilometer', u'mode': u'0640', u'owner': u'root', u'config_overrides': {u'DEFAULT': {u'exec_dirs': u'/openstack/venvs/ceilometer-19.1.0.dev14/bin,/sbin,/usr/sbin,/bin,/usr/bin,/usr/local/bin,/usr/local/sbin', u'filters_path': u'/etc/ceilometer/rootwrap.d,/usr/share/ceilometer/rootwrap'}}})
failed: [denver-compute01] (item={u'source_f': u'/etc/ceilometer/rootwrap.conf', u'tmp_f': u'/tmp/rootwrap.conf', u'config_type': u'ini', u'group': u'ceilometer', u'mode': u'0640', u'owner': u'root', u'config_overrides': {u'DEFAULT': {u'exec_dirs': u'/openstack/venvs/ceilometer-19.1.0.dev14/bin,/sbin,/usr/sbin,/bin,/usr/bin,/usr/local/bin,/usr/local/sbin', u'filters_path': u'/etc/ceilometer/rootwrap.d,/usr/share/ceilometer/rootwrap'}}}) => {"changed": true, "cmd": ["cp", "/etc/ceilometer/rootwrap.conf", "/etc/ceilometer/rootwrap.conf.original"], "delta": "0:00:00.004837", "end": "2019-04-10 14:40:23.291615", "item": {"config_overrides": {"DEFAULT": {"exec_dirs": "/openstack/venvs/ceilometer-19.1.0.dev14/bin,/sbin,/usr/sbin,/bin,/usr/bin,/usr/local/bin,/usr/local/sbin", "filters_path": "/etc/ceilometer/rootwrap.d,/usr/share/ceilometer/rootwrap"}}, "config_type": "ini", "group": "ceilometer", "mode": "0640", "owner": "root", "source_f": "/etc/ceilometer/rootwrap.conf", "tmp_f": "/tmp/rootwrap.conf"}, "msg": "non-zero return code", "rc": 1, "start": "2019-04-10 14:40:23.286778", "stderr": "cp: cannot stat '/etc/ceilometer/rootwrap.conf': No such file or directory", "stderr_lines": ["cp: cannot stat '/etc/ceilometer/rootwrap.conf': No such file or directory"], "stdout": "", "stdout_lines": []}
ok: [denver-infra01_ceilometer_central_container-9766bf53] => (item={u'source_f': u'/etc/ceilometer/polling.yaml', u'tmp_f': u'/tmp/polling.yaml', u'config_type': u'yaml', u'config_overrides': {}, u'list_extend': False, u'content': u''})
failed: [denver-compute01] (item={u'source_f': u'/etc/ceilometer/polling.yaml', u'tmp_f': u'/tmp/polling.yaml', u'config_type': u'yaml', u'config_overrides': {}, u'list_extend': False, u'content': u''}) => {"changed": true, "cmd": ["cp", "/etc/ceilometer/polling.yaml", "/etc/ceilometer/polling.yaml.original"], "delta": "0:00:00.004727", "end": "2019-04-10 14:40:23.422494", "item": {"config_overrides": {}, "config_type": "yaml", "content": "", "list_extend": false, "source_f": "/etc/ceilometer/polling.yaml", "tmp_f": "/tmp/polling.yaml"}, "msg": "non-zero return code", "rc": 1, "start": "2019-04-10 14:40:23.417767", "stderr": "cp: cannot stat '/etc/ceilometer/polling.yaml': No such file or directory", "stderr_lines": ["cp: cannot stat '/etc/ceilometer/polling.yaml': No such file or directory"], "stdout": "", "stdout_lines": []}
failed: [denver-compute01] (item={u'source_f': u'/etc/ceilometer/examples/loadbalancer_v2_meter_definitions.yaml', u'tmp_f': u'/tmp/loadbalancer_v2_meter_definitions.yaml', u'list_extend': False, u'target_f': u'/etc/ceilometer/loadbalancer_v2_meter_definitions.yaml', u'content': u'', u'config_type': u'yaml', u'config_overrides': {}, u'condition': True}) => {"changed": true, "cmd": ["cp", "/etc/ceilometer/examples/loadbalancer_v2_meter_definitions.yaml", "/etc/ceilometer/examples/loadbalancer_v2_meter_definitions.yaml.original"], "delta": "0:00:00.004772", "end": "2019-04-10 14:40:23.555779", "item": {"condition": true, "config_overrides": {}, "config_type": "yaml", "content": "", "list_extend": false, "source_f": "/etc/ceilometer/examples/loadbalancer_v2_meter_definitions.yaml", "target_f": "/etc/ceilometer/loadbalancer_v2_meter_definitions.yaml", "tmp_f": "/tmp/loadbalancer_v2_meter_definitions.yaml"}, "msg": "non-zero return code", "rc": 1, "start": "2019-04-10 14:40:23.551007", "stderr": "cp: cannot stat '/etc/ceilometer/examples/loadbalancer_v2_meter_definitions.yaml': No such file or directory", "stderr_lines": ["cp: cannot stat '/etc/ceilometer/examples/loadbalancer_v2_meter_definitions.yaml': No such file or directory"], "stdout": "", "stdout_lines": []}
ok: [denver-infra01_ceilometer_central_container-9766bf53] => (item={u'source_f': u'/etc/ceilometer/examples/loadbalancer_v2_meter_definitions.yaml', u'tmp_f': u'/tmp/loadbalancer_v2_meter_definitions.yaml', u'list_extend': False, u'target_f': u'/etc/ceilometer/loadbalancer_v2_meter_definitions.yaml', u'content': u'', u'config_type': u'yaml', u'config_overrides': {}, u'condition': True})
failed: [denver-compute01] (item={u'source_f': u'/etc/ceilometer/examples/osprofiler_event_definitions.yaml', u'tmp_f': u'/tmp/osprofiler_event_definitions.yaml', u'list_extend': False, u'target_f': u'/etc/ceilometer/osprofiler_event_definitions.yaml', u'content': u'', u'config_type': u'yaml', u'config_overrides': {}, u'condition': True}) => {"changed": true, "cmd": ["cp", "/etc/ceilometer/examples/osprofiler_event_definitions.yaml", "/etc/ceilometer/examples/osprofiler_event_definitions.yaml.original"], "delta": "0:00:00.004694", "end": "2019-04-10 14:40:23.686835", "item": {"condition": true, "config_overrides": {}, "config_type": "yaml", "content": "", "list_extend": false, "source_f": "/etc/ceilometer/examples/osprofiler_event_definitions.yaml", "target_f": "/etc/ceilometer/osprofiler_event_definitions.yaml", "tmp_f": "/tmp/osprofiler_event_definitions.yaml"}, "msg": "non-zero return code", "rc": 1, "start": "2019-04-10 14:40:23.682141", "stderr": "cp: cannot stat '/etc/ceilometer/examples/osprofiler_event_definitions.yaml': No such file or directory", "stderr_lines": ["cp: cannot stat '/etc/ceilometer/examples/osprofiler_event_definitions.yaml': No such file or directory"], "stdout": "", "stdout_lines": []}
failed: [denver-compute01] (item={u'source_f': u'/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/data/meters.d/meters.yaml', u'tmp_f': u'/tmp/meters.yaml', u'config_type': u'yaml', u'config_overrides': {}, u'list_extend': False, u'content': u''}) => {"changed": true, "cmd": ["cp", "/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/data/meters.d/meters.yaml", "/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/data/meters.d/meters.yaml.original"], "delta": "0:00:00.004396", "end": "2019-04-10 14:40:23.844326", "item": {"config_overrides": {}, "config_type": "yaml", "content": "", "list_extend": false, "source_f": "/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/data/meters.d/meters.yaml", "tmp_f": "/tmp/meters.yaml"}, "msg": "non-zero return code", "rc": 1, "start": "2019-04-10 14:40:23.839930", "stderr": "cp: cannot stat '/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/data/meters.d/meters.yaml': No such file or directory", "stderr_lines": ["cp: cannot stat '/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/data/meters.d/meters.yaml': No such file or directory"], "stdout": "", "stdout_lines": []}
ok: [denver-infra01_ceilometer_central_container-9766bf53] => (item={u'source_f': u'/etc/ceilometer/examples/osprofiler_event_definitions.yaml', u'tmp_f': u'/tmp/osprofiler_event_definitions.yaml', u'list_extend': False, u'target_f': u'/etc/ceilometer/osprofiler_event_definitions.yaml', u'content': u'', u'config_type': u'yaml', u'config_overrides': {}, u'condition': True})
failed: [denver-compute01] (item={u'source_f': u'/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/pipeline/data/event_definitions.yaml', u'tmp_f': u'/tmp/event_definitions.yaml', u'config_type': u'yaml', u'config_overrides': {}, u'list_extend': False, u'content': u''}) => {"changed": true, "cmd": ["cp", "/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/pipeline/data/event_definitions.yaml", "/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/pipeline/data/event_definitions.yaml.original"], "delta": "0:00:00.004467", "end": "2019-04-10 14:40:23.979521", "item": {"config_overrides": {}, "config_type": "yaml", "content": "", "list_extend": false, "source_f": "/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/pipeline/data/event_definitions.yaml", "tmp_f": "/tmp/event_definitions.yaml"}, "msg": "non-zero return code", "rc": 1, "start": "2019-04-10 14:40:23.975054", "stderr": "cp: cannot stat '/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/pipeline/data/event_definitions.yaml': No such file or directory", "stderr_lines": ["cp: cannot stat '/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/pipeline/data/event_definitions.yaml': No such file or directory"], "stdout": "", "stdout_lines": []}
ok: [denver-infra01_ceilometer_central_container-9766bf53] => (item={u'source_f': u'/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/data/meters.d/meters.yaml', u'tmp_f': u'/tmp/meters.yaml', u'config_type': u'yaml', u'config_overrides': {}, u'list_extend': False, u'content': u''})
failed: [denver-compute01] (item={u'source_f': u'/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/publisher/data/gnocchi_resources.yaml', u'tmp_f': u'/tmp/gnocchi_resources.yaml', u'config_type': u'yaml', u'config_overrides': {}, u'list_extend': False, u'content': u''}) => {"changed": true, "cmd": ["cp", "/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/publisher/data/gnocchi_resources.yaml", "/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/publisher/data/gnocchi_resources.yaml.original"], "delta": "0:00:00.004376", "end": "2019-04-10 14:40:24.103904", "item": {"config_overrides": {}, "config_type": "yaml", "content": "", "list_extend": false, "source_f": "/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/publisher/data/gnocchi_resources.yaml", "tmp_f": "/tmp/gnocchi_resources.yaml"}, "msg": "non-zero return code", "rc": 1, "start": "2019-04-10 14:40:24.099528", "stderr": "cp: cannot stat '/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/publisher/data/gnocchi_resources.yaml': No such file or directory", "stderr_lines": ["cp: cannot stat '/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/publisher/data/gnocchi_resources.yaml': No such file or directory"], "stdout": "", "stdout_lines": []}
ok: [denver-infra01_ceilometer_central_container-9766bf53] => (item={u'source_f': u'/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/pipeline/data/event_definitions.yaml', u'tmp_f': u'/tmp/event_definitions.yaml', u'config_type': u'yaml', u'config_overrides': {}, u'list_extend': False, u'content': u''})
ok: [denver-infra01_ceilometer_central_container-9766bf53] => (item={u'source_f': u'/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/publisher/data/gnocchi_resources.yaml', u'tmp_f': u'/tmp/gnocchi_resources.yaml', u'config_type': u'yaml', u'config_overrides': {}, u'list_extend': False, u'content': u''})

The failure occurs because the files were not originally copied due to failed task, and the task was skipped on subsequent runs due to the 'when' conditions here:

https://github.com/openstack/ansible-role-python_venv_build/blob/c94cd6c4b6efdec2352b5887a1b774d128997a6e/tasks/python_venv_install.yml#L112-L129

The same issue was observed with the Nova playbook under a different set of circumstances (repo container unreachable). After fixing that issue, the same tasks was skipped and similar failure points were seen.

To fix this, I ended up deleting the venv in each case and let the playbooks recreate.

Fix proposed to branch: master
Review: https://review.openstack.org/651598

Changed in openstack-ansible:
assignee: nobody → Mohammed Naser (mnaser)
status: New → In Progress
Mohammed Naser (mnaser) on 2019-04-10
Changed in openstack-ansible:
status: In Progress → Confirmed
importance: Undecided → Medium
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/651598
Committed: https://git.openstack.org/cgit/openstack/ansible-role-python_venv_build/commit/?id=b2abfe4bad7e7b440e79b048461cfbfda2b06757
Submitter: Zuul
Branch: master

commit b2abfe4bad7e7b440e79b048461cfbfda2b06757
Author: Mohammed Naser <email address hidden>
Date: Wed Apr 10 12:18:45 2019 -0400

    Delete constraints and requirements files on build fail

    If the installation of Python packages fails for any reason, the
    constraint and requirements file is already written and therefore
    unless you delete the virtualenv, it would not rebuild things
    again.

    This patch will handle failure and delete those files, then provide
    a warning for the user upon failing to inspect the needed file.

    Closes-Bug: #1824189

    Change-Id: I67db44ce10250908f23c26c90d271a780a875eb6

Changed in openstack-ansible:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/652161
Committed: https://git.openstack.org/cgit/openstack/ansible-role-python_venv_build/commit/?id=ed1e941038164649e79bca7e57c9e1d37a8fcfbe
Submitter: Zuul
Branch: stable/stein

commit ed1e941038164649e79bca7e57c9e1d37a8fcfbe
Author: Mohammed Naser <email address hidden>
Date: Wed Apr 10 12:18:45 2019 -0400

    Delete constraints and requirements files on build fail

    If the installation of Python packages fails for any reason, the
    constraint and requirements file is already written and therefore
    unless you delete the virtualenv, it would not rebuild things
    again.

    This patch will handle failure and delete those files, then provide
    a warning for the user upon failing to inspect the needed file.

    Closes-Bug: #1824189

    Change-Id: I67db44ce10250908f23c26c90d271a780a875eb6
    (cherry picked from commit b2abfe4bad7e7b440e79b048461cfbfda2b06757)

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers