Ceph integration in TripleO doesn't work with Ansible 2.6.1

[Emilien Macchi]

Testing TripleO with Ceph and Ansible 2.6.1, it fails during the "run nodes-uuid" task:

2018-07-25 20:17:25 | TASK [run nodes-uuid] **********************************************************
2018-07-25 20:17:25 | task path: /var/lib/mistral/ce840664-6f16-4c66-9179-1542e7987aaf/external_deploy_steps_tasks.yaml:169
2018-07-25 20:17:25 | Wednesday 25 July 2018 20:17:22 +0000 (0:00:00.079) 0:08:52.521 ********
2018-07-25 20:17:27 | fatal: [undercloud]: FAILED! => {"changed": true, "cmd": "ANSIBLE_LOG_PATH=\"/var/lib/mistral/ce840664-6f16-4c66-9179-1542e7987aaf/ceph-ansible/nodes_uuid_command.log\" ANSIBLE_SSH_RETRIES=3 ANSIBLE_HOST_KEY_CHECKING=False DEFAULT_FORKS=25 ansible-playbook --private-key /var/lib/mistral/ce840664-6f16-4c66-9179-1542e7987aaf/ssh_private_key -i /var/lib/mistral/ce840664-6f16-4c66-9179-1542e7987aaf/ceph-ansible/inventory.yml /var/lib/mistral/ce840664-6f16-4c66-9179-1542e7987aaf/ceph-ansible/nodes_uuid_playbook.yml", "delta": "0:00:02.858958", "end": "2018-07-25 20:17:25.324878", "msg": "non-zero return code", "rc": 4, "start": "2018-07-25 20:17:22.465920", "stderr": "", "stderr_lines": [], "stdout": "\nPLAY [all] *********************************************************************\n\nTASK [set nodes data] **********************************************************\nWednesday 25 July 2018 20:17:24 +0000 (0:00:00.179) 0:00:00.179 ******** \nok: [upstream-centos-7-rdo-cloud-0000073877]\n\nTASK [register machine id] *****************************************************\nWednesday 25 July 2018 20:17:24 +0000 (0:00:00.105) 0:00:00.284 ******** \nchanged: [upstream-centos-7-rdo-cloud-0000073877]\n\nTASK [generate host vars from nodes data] **************************************\nWednesday 25 July 2018 20:17:25 +0000 (0:00:00.433) 0:00:00.717 ******** \nfatal: [upstream-centos-7-rdo-cloud-0000073877]: UNREACHABLE! => {\"changed\": false, \"msg\": \"Authentication or permission failure. In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote tmp path in ansible.cfg to a path rooted in \\\"/tmp\\\". Failed command was: ( umask 77 && mkdir -p \\\"` echo /home/mistral/.ansible/tmp/ansible-tmp-1532549845.17-184540960422218 `\\\" && echo ansible-tmp-1Overcloud configuration failed.
2018-07-25 20:17:27 | END return value: 1
2018-07-25 20:17:27 | 532549845.17-184540960422218=\\\"` echo /home/mistral/.ansible/tmp/ansible-tmp-1532549845.17-184540960422218 `\\\" ), exited with result 1\", \"unreachable\": true}\n\nPLAY RECAP *********************************************************************\nupstream-centos-7-rdo-cloud-0000073877 : ok=2 changed=1 unreachable=1 failed=0 \n\nWednesday 25 July 2018 20:17:25 +0000 (0:00:00.104) 0:00:00.822 ******** \n=============================================================================== ", "stdout_lines": ["", "PLAY [all] *********************************************************************", "", "TASK [set nodes data] **********************************************************", "Wednesday 25 July 2018 20:17:24 +0000 (0:00:00.179) 0:00:00.179 ******** ", "ok: [upstream-centos-7-rdo-cloud-0000073877]", "", "TASK [register machine id] *****************************************************", "Wednesday 25 July 2018 20:17:24 +0000 (0:00:00.105) 0:00:00.284 ******** ", "changed: [upstream-centos-7-rdo-cloud-0000073877]", "", "TASK [generate host vars from nodes data] **************************************", "Wednesday 25 July 2018 20:17:25 +0000 (0:00:00.433) 0:00:00.717 ******** ", "fatal: [upstream-centos-7-rdo-cloud-0000073877]: UNREACHABLE! => {\"changed\": false, \"msg\": \"Authentication or permission failure. In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote tmp path in ansible.cfg to a path rooted in \\\"/tmp\\\". Failed command was: ( umask 77 && mkdir -p \\\"` echo /home/mistral/.ansible/tmp/ansible-tmp-1532549845.17-184540960422218 `\\\" && echo ansible-tmp-1532549845.17-184540960422218=\\\"` echo /home/mistral/.ansible/tmp/ansible-tmp-1532549845.17-184540960422218 `\\\" ), exited with result 1\", \"unreachable\": true}", "", "PLAY RECAP *********************************************************************", "upstream-centos-7-rdo-cloud-0000073877 : ok=2 changed=1 unreachable=1 failed=0 ", "", "Wednesday 25 July 2018 20:17:25 +0000 (0:00:00.104) 0:00:00.822 ******** ", "=============================================================================== "]}

https://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/docker/services/ceph-ansible/ceph-base.yaml#n418
https://logs.rdoproject.org/60/14960/5/check/legacy-rdoinfo-tripleo-master-testing-centos-7-multinode-1ctlr-featureset016/9f45eb5/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-07-25_20_17_27

Tested here: https://review.rdoproject.org/r/#/c/14960/

[John Fulton (jfulton-org)]

This should be fixed by https://review.openstack.org/#/c/583552

[John Fulton (jfulton-org)]

ceph-ansible master runs against ansible 2.6 already and 2.6 will probably be the excepted default in ceph-ansible 3.2.

Note that this bug isn't about ceph-ansible playbook though.

[John Fulton (jfulton-org)]

I predict that after https://review.openstack.org/#/c/583552 gets the test past the UUID playbook that ceph-ansible 3.1 will cause the test to fail later as per Seb:

ceph-ansible is configured to fail if > 2.5 on 3.1, so 2.6 will never work.

The simplest solution is to not move to ansible 2.6 until TripleO is using ceph-ansible 3.2. Does this cause a problem however?

[Giulio Fidente (gfidente)]

the latest version of ceph-ansible releases in CBS [1] ensures ansible >= 2.3 and does not enforce upper limits [2]

closing as resolved

1. http://cbs.centos.org/koji/buildinfo?buildID=24263
2. https://github.com/ceph/ceph-ansible/blob/v3.1.6/roles/ceph-common/tasks/checks/check_system.yml#L62-L73

[Giulio Fidente (gfidente)]

Yet it's not working with ansible 2.6 because of outdated version of config_template action plugin

For ceph-ansible to work with ansible 2.6 I think we need to resync the config_template action plugin with upstream, like it happened in the 3.2 branch with commit

https://github.com/ceph/ceph-ansible/commit/a1b3d5b7c3eed146372cbda5941653809504bb10

[Juan Antonio Osorio Robles (juan-osorio-robles)]

Is this still an issue?