OS: Ubuntu 18.04
DISTRIB_ID="OSA"
DISTRIB_RELEASE="19.1.0.dev14"
DISTRIB_CODENAME="Stein"
DISTRIB_DESCRIPTION="OpenStack-Ansible"
While deploying OSA, I found that if a playbook run fails or is cancelled prematurely, subsequent runs may also fail due to certain tasks being skipped based on 'when' conditions noted in this bug.
In this example, the 'Install python packages into the venv' task failed during the first run due to a lack of disk space:
TASK [python_venv_build : Install python packages into the venv] *********************************************************************
FAILED - RETRYING: Install python packages into the venv (5 retries left).
changed: [denver-infra01_ceilometer_central_container-9766bf53]
FAILED - RETRYING: Install python packages into the venv (4 retries left).
FAILED - RETRYING: Install python packages into the venv (3 retries left).
FAILED - RETRYING: Install python packages into the venv (2 retries left).
FAILED - RETRYING: Install python packages into the venv (1 retries left).
fatal: [denver-compute01]: FAILED! =>
...
Could not install packages due to an EnvironmentError: [Errno 28] No space left on device
After enlarging the disk, on a subsequent run, the task was skipped and the failure occurred here:
TASK [python_venv_build : Install python packages into the venv] *********************************************************************
skipping: [denver-infra01_ceilometer_central_container-9766bf53]
skipping: [denver-compute01]
...
TASK [os_ceilometer : Preserve original configuration file(s)] ***********************************************************************
[WARNING]: when statements should not include jinja2 templating delimiters such as {{ }} or {% %}. Found: {{ item.condition |
default(True) }}
[WARNING]: when statements should not include jinja2 templating delimiters such as {{ }} or {% %}. Found: {{ item.condition |
default(True) }}
ok: [denver-infra01_ceilometer_central_container-9766bf53] => (item={u'source_f': u'/etc/ceilometer/rootwrap.conf', u'tmp_f': u'/tmp/rootwrap.conf', u'config_type': u'ini', u'group': u'ceilometer', u'mode': u'0640', u'owner': u'root', u'config_overrides': {u'DEFAULT': {u'exec_dirs': u'/openstack/venvs/ceilometer-19.1.0.dev14/bin,/sbin,/usr/sbin,/bin,/usr/bin,/usr/local/bin,/usr/local/sbin', u'filters_path': u'/etc/ceilometer/rootwrap.d,/usr/share/ceilometer/rootwrap'}}})
failed: [denver-compute01] (item={u'source_f': u'/etc/ceilometer/rootwrap.conf', u'tmp_f': u'/tmp/rootwrap.conf', u'config_type': u'ini', u'group': u'ceilometer', u'mode': u'0640', u'owner': u'root', u'config_overrides': {u'DEFAULT': {u'exec_dirs': u'/openstack/venvs/ceilometer-19.1.0.dev14/bin,/sbin,/usr/sbin,/bin,/usr/bin,/usr/local/bin,/usr/local/sbin', u'filters_path': u'/etc/ceilometer/rootwrap.d,/usr/share/ceilometer/rootwrap'}}}) => {"changed": true, "cmd": ["cp", "/etc/ceilometer/rootwrap.conf", "/etc/ceilometer/rootwrap.conf.original"], "delta": "0:00:00.004837", "end": "2019-04-10 14:40:23.291615", "item": {"config_overrides": {"DEFAULT": {"exec_dirs": "/openstack/venvs/ceilometer-19.1.0.dev14/bin,/sbin,/usr/sbin,/bin,/usr/bin,/usr/local/bin,/usr/local/sbin", "filters_path": "/etc/ceilometer/rootwrap.d,/usr/share/ceilometer/rootwrap"}}, "config_type": "ini", "group": "ceilometer", "mode": "0640", "owner": "root", "source_f": "/etc/ceilometer/rootwrap.conf", "tmp_f": "/tmp/rootwrap.conf"}, "msg": "non-zero return code", "rc": 1, "start": "2019-04-10 14:40:23.286778", "stderr": "cp: cannot stat '/etc/ceilometer/rootwrap.conf': No such file or directory", "stderr_lines": ["cp: cannot stat '/etc/ceilometer/rootwrap.conf': No such file or directory"], "stdout": "", "stdout_lines": []}
ok: [denver-infra01_ceilometer_central_container-9766bf53] => (item={u'source_f': u'/etc/ceilometer/polling.yaml', u'tmp_f': u'/tmp/polling.yaml', u'config_type': u'yaml', u'config_overrides': {}, u'list_extend': False, u'content': u''})
failed: [denver-compute01] (item={u'source_f': u'/etc/ceilometer/polling.yaml', u'tmp_f': u'/tmp/polling.yaml', u'config_type': u'yaml', u'config_overrides': {}, u'list_extend': False, u'content': u''}) => {"changed": true, "cmd": ["cp", "/etc/ceilometer/polling.yaml", "/etc/ceilometer/polling.yaml.original"], "delta": "0:00:00.004727", "end": "2019-04-10 14:40:23.422494", "item": {"config_overrides": {}, "config_type": "yaml", "content": "", "list_extend": false, "source_f": "/etc/ceilometer/polling.yaml", "tmp_f": "/tmp/polling.yaml"}, "msg": "non-zero return code", "rc": 1, "start": "2019-04-10 14:40:23.417767", "stderr": "cp: cannot stat '/etc/ceilometer/polling.yaml': No such file or directory", "stderr_lines": ["cp: cannot stat '/etc/ceilometer/polling.yaml': No such file or directory"], "stdout": "", "stdout_lines": []}
failed: [denver-compute01] (item={u'source_f': u'/etc/ceilometer/examples/loadbalancer_v2_meter_definitions.yaml', u'tmp_f': u'/tmp/loadbalancer_v2_meter_definitions.yaml', u'list_extend': False, u'target_f': u'/etc/ceilometer/loadbalancer_v2_meter_definitions.yaml', u'content': u'', u'config_type': u'yaml', u'config_overrides': {}, u'condition': True}) => {"changed": true, "cmd": ["cp", "/etc/ceilometer/examples/loadbalancer_v2_meter_definitions.yaml", "/etc/ceilometer/examples/loadbalancer_v2_meter_definitions.yaml.original"], "delta": "0:00:00.004772", "end": "2019-04-10 14:40:23.555779", "item": {"condition": true, "config_overrides": {}, "config_type": "yaml", "content": "", "list_extend": false, "source_f": "/etc/ceilometer/examples/loadbalancer_v2_meter_definitions.yaml", "target_f": "/etc/ceilometer/loadbalancer_v2_meter_definitions.yaml", "tmp_f": "/tmp/loadbalancer_v2_meter_definitions.yaml"}, "msg": "non-zero return code", "rc": 1, "start": "2019-04-10 14:40:23.551007", "stderr": "cp: cannot stat '/etc/ceilometer/examples/loadbalancer_v2_meter_definitions.yaml': No such file or directory", "stderr_lines": ["cp: cannot stat '/etc/ceilometer/examples/loadbalancer_v2_meter_definitions.yaml': No such file or directory"], "stdout": "", "stdout_lines": []}
ok: [denver-infra01_ceilometer_central_container-9766bf53] => (item={u'source_f': u'/etc/ceilometer/examples/loadbalancer_v2_meter_definitions.yaml', u'tmp_f': u'/tmp/loadbalancer_v2_meter_definitions.yaml', u'list_extend': False, u'target_f': u'/etc/ceilometer/loadbalancer_v2_meter_definitions.yaml', u'content': u'', u'config_type': u'yaml', u'config_overrides': {}, u'condition': True})
failed: [denver-compute01] (item={u'source_f': u'/etc/ceilometer/examples/osprofiler_event_definitions.yaml', u'tmp_f': u'/tmp/osprofiler_event_definitions.yaml', u'list_extend': False, u'target_f': u'/etc/ceilometer/osprofiler_event_definitions.yaml', u'content': u'', u'config_type': u'yaml', u'config_overrides': {}, u'condition': True}) => {"changed": true, "cmd": ["cp", "/etc/ceilometer/examples/osprofiler_event_definitions.yaml", "/etc/ceilometer/examples/osprofiler_event_definitions.yaml.original"], "delta": "0:00:00.004694", "end": "2019-04-10 14:40:23.686835", "item": {"condition": true, "config_overrides": {}, "config_type": "yaml", "content": "", "list_extend": false, "source_f": "/etc/ceilometer/examples/osprofiler_event_definitions.yaml", "target_f": "/etc/ceilometer/osprofiler_event_definitions.yaml", "tmp_f": "/tmp/osprofiler_event_definitions.yaml"}, "msg": "non-zero return code", "rc": 1, "start": "2019-04-10 14:40:23.682141", "stderr": "cp: cannot stat '/etc/ceilometer/examples/osprofiler_event_definitions.yaml': No such file or directory", "stderr_lines": ["cp: cannot stat '/etc/ceilometer/examples/osprofiler_event_definitions.yaml': No such file or directory"], "stdout": "", "stdout_lines": []}
failed: [denver-compute01] (item={u'source_f': u'/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/data/meters.d/meters.yaml', u'tmp_f': u'/tmp/meters.yaml', u'config_type': u'yaml', u'config_overrides': {}, u'list_extend': False, u'content': u''}) => {"changed": true, "cmd": ["cp", "/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/data/meters.d/meters.yaml", "/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/data/meters.d/meters.yaml.original"], "delta": "0:00:00.004396", "end": "2019-04-10 14:40:23.844326", "item": {"config_overrides": {}, "config_type": "yaml", "content": "", "list_extend": false, "source_f": "/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/data/meters.d/meters.yaml", "tmp_f": "/tmp/meters.yaml"}, "msg": "non-zero return code", "rc": 1, "start": "2019-04-10 14:40:23.839930", "stderr": "cp: cannot stat '/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/data/meters.d/meters.yaml': No such file or directory", "stderr_lines": ["cp: cannot stat '/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/data/meters.d/meters.yaml': No such file or directory"], "stdout": "", "stdout_lines": []}
ok: [denver-infra01_ceilometer_central_container-9766bf53] => (item={u'source_f': u'/etc/ceilometer/examples/osprofiler_event_definitions.yaml', u'tmp_f': u'/tmp/osprofiler_event_definitions.yaml', u'list_extend': False, u'target_f': u'/etc/ceilometer/osprofiler_event_definitions.yaml', u'content': u'', u'config_type': u'yaml', u'config_overrides': {}, u'condition': True})
failed: [denver-compute01] (item={u'source_f': u'/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/pipeline/data/event_definitions.yaml', u'tmp_f': u'/tmp/event_definitions.yaml', u'config_type': u'yaml', u'config_overrides': {}, u'list_extend': False, u'content': u''}) => {"changed": true, "cmd": ["cp", "/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/pipeline/data/event_definitions.yaml", "/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/pipeline/data/event_definitions.yaml.original"], "delta": "0:00:00.004467", "end": "2019-04-10 14:40:23.979521", "item": {"config_overrides": {}, "config_type": "yaml", "content": "", "list_extend": false, "source_f": "/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/pipeline/data/event_definitions.yaml", "tmp_f": "/tmp/event_definitions.yaml"}, "msg": "non-zero return code", "rc": 1, "start": "2019-04-10 14:40:23.975054", "stderr": "cp: cannot stat '/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/pipeline/data/event_definitions.yaml': No such file or directory", "stderr_lines": ["cp: cannot stat '/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/pipeline/data/event_definitions.yaml': No such file or directory"], "stdout": "", "stdout_lines": []}
ok: [denver-infra01_ceilometer_central_container-9766bf53] => (item={u'source_f': u'/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/data/meters.d/meters.yaml', u'tmp_f': u'/tmp/meters.yaml', u'config_type': u'yaml', u'config_overrides': {}, u'list_extend': False, u'content': u''})
failed: [denver-compute01] (item={u'source_f': u'/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/publisher/data/gnocchi_resources.yaml', u'tmp_f': u'/tmp/gnocchi_resources.yaml', u'config_type': u'yaml', u'config_overrides': {}, u'list_extend': False, u'content': u''}) => {"changed": true, "cmd": ["cp", "/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/publisher/data/gnocchi_resources.yaml", "/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/publisher/data/gnocchi_resources.yaml.original"], "delta": "0:00:00.004376", "end": "2019-04-10 14:40:24.103904", "item": {"config_overrides": {}, "config_type": "yaml", "content": "", "list_extend": false, "source_f": "/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/publisher/data/gnocchi_resources.yaml", "tmp_f": "/tmp/gnocchi_resources.yaml"}, "msg": "non-zero return code", "rc": 1, "start": "2019-04-10 14:40:24.099528", "stderr": "cp: cannot stat '/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/publisher/data/gnocchi_resources.yaml': No such file or directory", "stderr_lines": ["cp: cannot stat '/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/publisher/data/gnocchi_resources.yaml': No such file or directory"], "stdout": "", "stdout_lines": []}
ok: [denver-infra01_ceilometer_central_container-9766bf53] => (item={u'source_f': u'/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/pipeline/data/event_definitions.yaml', u'tmp_f': u'/tmp/event_definitions.yaml', u'config_type': u'yaml', u'config_overrides': {}, u'list_extend': False, u'content': u''})
ok: [denver-infra01_ceilometer_central_container-9766bf53] => (item={u'source_f': u'/openstack/venvs/ceilometer-19.1.0.dev14/lib/python2.7/site-packages/ceilometer/publisher/data/gnocchi_resources.yaml', u'tmp_f': u'/tmp/gnocchi_resources.yaml', u'config_type': u'yaml', u'config_overrides': {}, u'list_extend': False, u'content': u''})
The failure occurs because the files were not originally copied due to failed task, and the task was skipped on subsequent runs due to the 'when' conditions here:
https://github.com/openstack/ansible-role-python_venv_build/blob/c94cd6c4b6efdec2352b5887a1b774d128997a6e/tasks/python_venv_install.yml#L112-L129
The same issue was observed with the Nova playbook under a different set of circumstances (repo container unreachable). After fixing that issue, the same tasks was skipped and similar failure points were seen.
To fix this, I ended up deleting the venv in each case and let the playbooks recreate.
Fix proposed to branch: master
Review: https:/