During upgrade Stein > Train at the time of running the converge step (executes the whole deployment underneath) the following task failed:
2020-04-07 14:40:05,075 p=48174 u=mistral | TASK [tripleo-ceph-common : set calling_ansible_environment_variables] *********
2020-04-07 14:40:05,076 p=48174 u=mistral | Tuesday 07 April 2020 14:40:05 +0000 (0:00:00.134) 0:06:57.979 *********
2020-04-07 14:40:05,134 p=48174 u=mistral | skipping: [undercloud] => {"changed": false, "skip_reason": "Conditional result was False"}
2020-04-07 14:40:05,194 p=48174 u=mistral | TASK [create ceph-ansible working direcotry] ***********************************
2020-04-07 14:40:05,195 p=48174 u=mistral | Tuesday 07 April 2020 14:40:05 +0000 (0:00:00.118) 0:06:58.098 *********
2020-04-07 14:40:05,470 p=48174 u=mistral | TASK [tripleo-ceph-work-dir : create ceph-ansible temp dirs] *******************
2020-04-07 14:40:05,471 p=48174 u=mistral | Tuesday 07 April 2020 14:40:05 +0000 (0:00:00.275) 0:06:58.374 *********
2020-04-07 14:40:05,832 p=48174 u=mistral | changed: [undercloud] => (item=/var/lib/mistral/qe-Cloud-0/ceph-ansible) => {"ansible_loop_var": "item", "changed": true, "gid": 42430, "group": "42430", "item": "/var/lib/mistral/qe-Cloud-0/ceph-ansible", "mode": "0755", "owner": "tripleo-admin", "path": "/var/lib/mistral/qe-Cloud-0/ceph-ansible", "secontext": "system_u:object_r:container_file_t:s0", "size": 205, "state": "directory", "uid": 1002}
2020-04-07 14:40:06,137 p=48174 u=mistral | changed: [undercloud] => (item=/var/lib/mistral/qe-Cloud-0/ceph-ansible/group_vars) => {"ansible_loop_var": "item", "changed": true, "gid": 42430, "group": "42430", "item": "/var/lib/mistral/qe-Cloud-0/ceph-ansible/group_vars", "mode": "0755", "owner": "tripleo-admin", "path": "/var/lib/mistral/qe-Cloud-0/ceph-ansible/group_vars", "secontext": "system_u:object_r:container_file_t:s0", "size": 88, "state": "directory", "uid": 1002}
2020-04-07 14:40:06,390 p=48174 u=mistral | changed: [undercloud] => (item=/var/lib/mistral/qe-Cloud-0/ceph-ansible/host_vars) => {"ansible_loop_var": "item", "changed": true, "gid": 42430, "group": "42430", "item": "/var/lib/mistral/qe-Cloud-0/ceph-ansible/host_vars", "mode": "0755", "owner": "tripleo-admin", "path": "/var/lib/mistral/qe-Cloud-0/ceph-ansible/host_vars", "secontext": "system_u:object_r:container_file_t:s0", "size": 174, "state": "directory", "uid": 1002}
2020-04-07 14:40:06,646 p=48174 u=mistral | changed: [undercloud] => (item=/var/lib/mistral/qe-Cloud-0/ceph-ansible/fetch_dir) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/var/lib/mistral/qe-Cloud-0/ceph-ansible/fetch_dir", "mode": "0755", "owner": "tripleo-admin", "path": "/var/lib/mistral/qe-Cloud-0/ceph-ansible/fetch_dir", "secontext": "unconfined_u:object_r:container_file_t:s0", "size": 6, "state": "directory", "uid": 1002}
2020-04-07 14:40:06,729 p=48174 u=mistral | TASK [tripleo-ceph-work-dir : symbolic link to tripleo inventory from ceph-ansible work directory] ***
2020-04-07 14:40:06,729 p=48174 u=mistral | Tuesday 07 April 2020 14:40:06 +0000 (0:00:01.258) 0:06:59.632 *********
2020-04-07 14:40:07,040 p=48174 u=mistral | fatal: [undercloud]: FAILED! => {"changed": false, "gid": 42430, "group": "42430", "mode": "0664", "msg": "refusing to convert from file to symlink for /var/lib/mistral/qe-Cloud-0/ceph-ansible/inventory.yml", "owner": "42430", "path": "/var/lib/mistral/qe-Cloud-0/ceph-ansible/inventory.yml", "secontext": "system_u:object_r:container_file_t:s0", "size": 1303, "state": "file", "uid": 42430}
2020-04-07 14:40:07,041 p=48174 u=mistral | NO MORE HOSTS LEFT *************************************************************
2020-04-07 14:40:07,042 p=48174 u=mistral | PLAY RECAP *********************************************************************
2020-04-07 14:40:07,042 p=48174 u=mistral | ceph-0 : ok=103 changed=21 unreachable=0 failed=0 skipped=55 rescued=0 ignored=0
2020-04-07 14:40:07,043 p=48174 u=mistral | ceph-1 : ok=99 changed=21 unreachable=0 failed=0 skipped=55 rescued=0 ignored=0
2020-04-07 14:40:07,043 p=48174 u=mistral | ceph-2 : ok=99 changed=21 unreachable=0 failed=0 skipped=55 rescued=0 ignored=0
2020-04-07 14:40:07,043 p=48174 u=mistral | compute-0 : ok=128 changed=37 unreachable=0 failed=0 skipped=71 rescued=0 ignored=0
2020-04-07 14:40:07,043 p=48174 u=mistral | compute-1 : ok=127 changed=37 unreachable=0 failed=0 skipped=71 rescued=0 ignored=0
2020-04-07 14:40:07,043 p=48174 u=mistral | controller-0 : ok=151 changed=51 unreachable=0 failed=0 skipped=73 rescued=0 ignored=0
2020-04-07 14:40:07,043 p=48174 u=mistral | controller-1 : ok=148 changed=50 unreachable=0 failed=0 skipped=73 rescued=0 ignored=0
2020-04-07 14:40:07,043 p=48174 u=mistral | controller-2 : ok=148 changed=50 unreachable=0 failed=0 skipped=73 rescued=0 ignored=0
2020-04-07 14:40:07,044 p=48174 u=mistral | undercloud : ok=12 changed=3 unreachable=0 failed=1 skipped=4 rescued=0 ignored=0
2020-04-07 14:40:07,044 p=48174 u=mistral | Tuesday 07 April 2020 14:40:07 +0000 (0:00:00.314) 0:06:59.947 *********
2020-04-07 14:40:07,044 p=48174 u=mistral | ===============================================================================
After some debugging and local testing, I did try adding force: yes to the task:
- name: symbolic link to tripleo inventory from ceph-ansible work directory
# If we call ceph-ansible with the same inventory as the calling
# playbook, then config-download/groups_vars will be used instead
# of config-download/ceph-ansible/group_vars.
file:
src: "{{ inventory_file }}"
dest: "{{ playbook_dir }}/ceph-ansible/inventory.yml"
state: link
force: true
tags:
- run_uuid_ansible
- run_ceph_ansible
Re-ran the same converge and then the task successfuly executed:
TASK [tripleo-ceph-work-dir : create ceph-ansible temp dirs] *******************
Tuesday 07 April 2020 16:14:32 +0000 (0:00:00.254) 0:07:43.964 *********
ok: [undercloud] => (item=/var/lib/mistral/qe-Cloud-0/ceph-ansible) => {"ansible_loop_var": "item", "changed": false, "gid": 42430, "group": "42430", "item": "/var/lib/mistral/qe-Cloud-0/ceph-ansible", "mode": "0755", "owner": "tripleo-admin", "path": "/var/lib/mistral/qe-Cloud-0/ceph-ansible", "secontext": "system_u:object_r:container_file_t:s0", "size": 222, "state": "directory", "uid": 1002}
ok: [undercloud] => (item=/var/lib/mistral/qe-Cloud-0/ceph-ansible/group_vars) => {"ansible_loop_var": "item", "changed": false, "gid": 42430, "group": "42430", "item": "/var/lib/mistral/qe-Cloud-0/ceph-ansible/group_vars", "mode": "0755", "owner": "tripleo-admin", "path": "/var/lib/mistral/qe-Cloud-0/ceph-ansible/group_vars", "secontext": "system_u:object_r:container_file_t:s0", "size": 88, "state": "directory", "uid": 1002}
ok: [undercloud] => (item=/var/lib/mistral/qe-Cloud-0/ceph-ansible/host_vars) => {"ansible_loop_var": "item", "changed": false, "gid": 42430, "group": "42430", "item": "/var/lib/mistral/qe-Cloud-0/ceph-ansible/host_vars", "mode": "0755", "owner": "tripleo-admin", "path": "/var/lib/mistral/qe-Cloud-0/ceph-ansible/host_vars", "secontext": "system_u:object_r:container_file_t:s0", "size": 174, "state": "directory", "uid": 1002}
ok: [undercloud] => (item=/var/lib/mistral/qe-Cloud-0/ceph-ansible/fetch_dir) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/var/lib/mistral/qe-Cloud-0/ceph-ansible/fetch_dir", "mode": "0755", "owner": "tripleo-admin", "path": "/var/lib/mistral/qe-Cloud-0/ceph-ansible/fetch_dir", "secontext": "unconfined_u:object_r:container_file_t:s0", "size": 6, "state": "directory", "uid": 1002}
TASK [tripleo-ceph-work-dir : symbolic link to tripleo inventory from ceph-ansible work directory] ***
Tuesday 07 April 2020 16:14:33 +0000 (0:00:01.312) 0:07:45.277 *********
changed: [undercloud] => {"changed": true, "dest": "/var/lib/mistral/qe-Cloud-0/ceph-ansible/inventory.yml", "gid": 1003, "group": "tripleo-admin", "mode": "0777", "owner": "tripleo-admin", "secontext": "unconfined_u:object_r:container_file_t:s0", "size": 58, "src": "/var/lib/mistral/qe-Cloud-0/tripleo-ansible-inventory.yaml", "state": "link", "uid": 1002}
Checking the upstreame code, it looks like that line was missed to backport to tripleo-ansible stable/train: https://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/roles/tripleo_ceph_work_dir/tasks/prepare.yml#L37
'force: true' is not present in train [1] but it is present in master [2].
Thus the patch will need to go directly into train.
[1] https:/
[2] https:/