Unable to find 'Undercloud/deployment-hierarchy.yaml'

[Cédric Jeanneret]

We get failures in the CI with the following message:

2019-06-24 11:44:22 | TASK [tripleo-hieradata : Write hiera config] **********************************
2019-06-24 11:44:22 | [WARNING]: Unable to find 'Undercloud/deployment-hierarchy.yaml' in expected
2019-06-24 11:44:22 | paths (use -vvvvv to see paths)
2019-06-24 11:44:22 | fatal: [undercloud]: FAILED! => {"msg": "The conditional check 'hieradata_files != []' failed. The error was: An unhandled exception occurred while templating '{{ lookup('file', 'Undercloud/deployment-hierarchy.yaml') }}'. Error was a <class 'ansible.errors.AnsibleError'>, original message: An unhandled exception occurred while running the lookup plugin 'file'. Error was a <class 'ansible.errors.AnsibleError'>, original message: could not locate file in lookup: Undercloud/deployment-hierarchy.yaml\n\nThe error appears to have been in '/usr/share/ansible/roles/tripleo-hieradata/tasks/main.yaml': line 7, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Write hiera config\n ^ here\n"}
2019-06-24 11:44:22 |
2019-06-24 11:44:22 | NO MORE HOSTS LEFT *************************************************************
2019-06-24 11:44:22 |
2019-06-24 11:44:22 | PLAY RECAP *********************************************************************
2019-06-24 11:44:22 | undercloud : ok=24 changed=7 unreachable=0 failed=1
2019-06-24 11:44:22 |
2019-06-24 11:44:22 | Exception: Deployment failed
2019-06-24 11:44:22 | Traceback (most recent call last):
2019-06-24 11:44:22 | File "/usr/lib/python2.7/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1253, in _standalone_deploy
2019-06-24 11:44:22 | raise exceptions.DeploymentError('Deployment failed')
2019-06-24 11:44:22 | DeploymentError: Deployment failed
2019-06-24 11:44:22 | None
2019-06-24 11:44:24 | ** Found ansible errors for undercloud deployment! **
2019-06-24 11:44:24 | [
2019-06-24 11:44:24 | [
2019-06-24 11:44:24 | "Write hiera config",
2019-06-24 11:44:24 | {
2019-06-24 11:44:24 | "msg": "The conditional check 'hieradata_files != []' failed. The error was: An unhandled exception occurred while templating '{{ lookup('file', 'Undercloud/deployment-hierarchy.yaml') }}'. Error was a <class 'ansible.errors.AnsibleError'>, original message: An unhandled exception occurred while running the lookup plugin 'file'. Error was a <class 'ansible.errors.AnsibleError'>, original message: could not locate file in lookup: Undercloud/deployment-hierarchy.yaml\n\nThe error appears to have been in '/usr/share/ansible/roles/tripleo-hieradata/tasks/main.yaml': line 7, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Write hiera config\n ^ here\n",
2019-06-24 11:44:24 | "_ansible_no_log": false
2019-06-24 11:44:24 | }
2019-06-24 11:44:24 | ]
2019-06-24 11:44:24 | ]
2019-06-24 11:44:24 | Not cleaning working directory /home/zuul/tripleo-heat-installer-templates
2019-06-24 11:44:24 | Not cleaning ansible directory /home/zuul/undercloud-ansible-ED7iQa
2019-06-24 11:44:24 | Install artifact is located at /home/zuul/undercloud-install-20190624114422.tar.bzip2
2019-06-24 11:44:24 | The heat stack undercloud virtual state/action is reset to CREATE. Use "--force-stack-update" to set it forcefully to UPDATE
2019-06-24 11:44:24 | Removing the stack virtual update mark file /var/lib/tripleo-heat-installer/update_mark_undercloud
2019-06-24 11:44:24 |
2019-06-24 11:44:24 | !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2019-06-24 11:44:24 |
2019-06-24 11:44:24 | Deployment Failed!

It seems to be linked to the following changes:
- https://review.opendev.org/#/c/662562/
- https://review.opendev.org/#/c/662531/ (likely the culprit)

We have 48 hits in the last 24 hours:
http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22The%20conditional%20check%20'hieradata_files%20!%3D%20%5B%5D'%20failed%5C%22

Cheers,

C.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/667104

[Cédric Jeanneret (cjeanner)]

So the failure is:
https://review.opendev.org/#/c/662562/20/common/deploy-steps.j2@411 introduce a file lookup, and that file doesn't exist, at least for an upgrade (since deploy works fine)

that lookup reports the error here: https://review.opendev.org/#/c/662531/16/roles/tripleo-hieradata/tasks/main.yaml@8 with that cryptic thing about Ansible object.

Wondering if the change introduced by Emilien didn't overlook something regarding upgrade in the deploy steps or something.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Emilien Macchi (<email address hidden>) on branch: master
Review: https://review.opendev.org/667344