Unreachable overcloud nodes during "run nodes-uuid" task

[John Fulton]

TripleO Ceph installation using external deploy tasks with the following:

TASK [run nodes-uuid] **********************************************************
Friday 13 July 2018 20:56:52 -0400 (0:00:00.039) 0:10:15.964 ***********
fatal: [undercloud]: FAILED! => {"changed": true, "cmd": "ANSIBLE_LOG_PATH=\"/var/lib/mistral/dab7ef10-b96d-44c4-a639-4270c8a6d019/ceph-ansible/nodes_uuid_command.log\" ANSIBLE_SSH_RETRIES=3 ANSIBLE_HOST_KEY_CHECKING=False DEFAULT_FORKS=25 ansible-playbook --private-key /var/lib/mistral/dab7ef10-b96d-44c4-a639-4270c8a6d019/ssh_private_key -i /var/lib/mistral/dab7ef10-b96d-44c4-a639-4270c8a6d019/ceph-ansible/inventory.yml /var/lib/mistral/dab7ef10-b96d-44c4-a639-4270c8a6d019/ceph-ansible/nodes_uuid_playbook.yml", "delta": "0:00:01.307249", "end": "2018-07-13 20:56:54.254304", "msg": "non-zero return code", "rc": 4, "start": "2018-07-13 20:56:52.947055", "stderr": "", "stderr_lines": [], "stdout": "\nPLAY [all] *********************************************************************\n\nTASK [set nodes data] **********************************************************\nFriday 13 July 2018 20:56:53 -0400 (0:00:00.068) 0:00:00.069 *********** \nok: [compute-0]\nok: [ceph-0]\nok: [controller-0]\n\nTASK [register machine id] *****************************************************\nFriday 13 July 2018 20:56:53 -0400 (0:00:00.066) 0:00:00.135 *********** \nchanged: [ceph-0]\nchanged: [controller-0]\nchanged: [compute-0]\n\nTASK [generate host vars from nodes data] **************************************\nFriday 13 July 2018 20:56:54 -0400 (0:00:00.510) 0:00:00.645 *********** \nfatal: [compute-0]: UNREACHABLE! => {\"changed\": false, \"msg\": \"Authentication or permission failure. In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote tmp path in ansible.cfg to a path rooted in \\\"/tmp\\\". Failed command was: ( umask 77 && mkdir -p \\\"` echo /home/mistral/.ansible/tmp/ansible-tmp-1531529814.17-222746662763579 `\\\" && echo ansible-tmp-1531529814.17-222746662763579=\\\"` echo /home/mistral/.ansible/tmp/ansible-tmp-1531529814.17-222746662763579 `\\\" ), exited with result 1\", \"unreachable\": true}\nfatal: [ceph-0]: UNREACHABLE! => {\"changed\": false, \"msg\": \"Authentication or permission failure. In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote tmp path in ansible.cfg to a path rooted in \\\"/tmp\\\". Failed command was: ( umask 77 && mkdir -p \\\"` echo /home/mistral/.ansible/tmp/ansible-tmp-1531529814.18-201209661451782 `\\\" && echo ansible-tmp-1531529814.18-201209661451782=\\\"` echo /home/mistral/.ansible/tmp/ansible-tmp-1531529814.18-201209661451782 `\\\" ), exited with result 1\", \"unreachable\": true}\nfatal: [controller-0]: UNREACHABLE! => {\"changed\": false, \"msg\": \"Authentication or permission failure. In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote tmp path in ansible.cfg to a path rooted in \\\"/tmp\\\". Failed command was: ( umask 77 && mkdir -p \\\"` echo /home/mistral/.ansible/tmp/ansible-tmp-1531529814.19-31199768599421 `\\\" && echo ansible-tmp-1531529814.19-31199768599421=\\\"` echo /home/mistral/.ansible/tmp/ansible-tmp-1531529814.19-31199768599421 `\\\" ), exited with result 1\", \"unreachable\": true}\n\nPLAY RECAP *********************************************************************\nceph-0 : ok=2 changed=1 unreachable=1 failed=0 \ncompute-0 : ok=2 changed=1 unreachable=1 failed=0 \ncontroller-0 : ok=2 changed=1 unreachable=1 failed=0 \n\nFriday 13 July 2018 20:56:54 -0400 (0:00:00.068) 0:00:00.713 ***********
...
NO MORE HOSTS LEFT *************************************************************
PLAY RECAP *********************************************************************
ceph-0 : ok=89 changed=42 unreachable=0 failed=0
compute-0 : ok=107 changed=44 unreachable=0 failed=0
controller-0 : ok=147 changed=45 unreachable=0 failed=0
undercloud : ok=19 changed=10 unreachable=0 failed=1
Friday 13 July 2018 20:56:54 -0400 (0:00:01.509) 0:10:17.474 ***********
===============================================================================

also, {{playbook_dir}}/ceph-ansible/nodes_uuid_command.log reports:

TASK [generate host vars from nodes data] **************************************
Friday 13 July 2018 20:56:54 -0400 (0:00:00.510) 0:00:00.645 ***********
fatal: [compute-0]: UNREACHABLE! => {"changed": false, "msg": "Authentication or permission failure. In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote tmp path in ansible.cfg to a path rooted in \"/tmp\". Failed command was: ( umask 77 && mkdir -p \"` echo /home/mistral/.ansible/tmp/ansible-tmp-1531529814.17-222746662763579 `\" && echo ansible-tmp-1531529814.17-222746662763579=\"` echo /home/mistral/.ansible/tmp/ansible-tmp-1531529814.17-222746662763579 `\" ), exited with result 1", "unreachable": true}
fatal: [ceph-0]: UNREACHABLE! => {"changed": false, "msg": "Authentication or permission failure. In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote tmp path in ansible.cfg to a path rooted in \"/tmp\". Failed command was: ( umask 77 && mkdir -p \"` echo /home/mistral/.ansible/tmp/ansible-tmp-1531529814.18-201209661451782 `\" && echo ansible-tmp-1531529814.18-201209661451782=\"` echo /home/mistral/.ansible/tmp/ansible-tmp-1531529814.18-201209661451782 `\" ), exited with result 1", "unreachable": true}
fatal: [controller-0]: UNREACHABLE! => {"changed": false, "msg": "Authentication or permission failure. In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote tmp path in ansible.cfg to a path rooted in \"/tmp\". Failed command was: ( umask 77 && mkdir -p \"` echo /home/mistral/.ansible/tmp/ansible-tmp-1531529814.19-31199768599421 `\" && echo ansible-tmp-1531529814.19-31199768599421=\"` echo /home/mistral/.ansible/tmp/ansible-tmp-1531529814.19-31199768599421 `\" ), exited with result 1", "unreachable": true}
PLAY RECAP *********************************************************************
ceph-0 : ok=2 changed=1 unreachable=1 failed=0
compute-0 : ok=2 changed=1 unreachable=1 failed=0
controller-0 : ok=2 changed=1 unreachable=1 failed=0
Friday 13 July 2018 20:56:54 -0400 (0:00:00.068) 0:00:00.713 ***********
===============================================================================

[John Fulton (jfulton-org)]

fix at: https://review.openstack.org/#/c/583552/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/583552
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=694ac8ce46aa21559e8e8f6800789d88fadb8844
Submitter: Zuul
Branch: master

commit 694ac8ce46aa21559e8e8f6800789d88fadb8844
Author: Giulio Fidente <email address hidden>
Date: Wed Jul 18 13:39:07 2018 +0200

    Use global ansible.cfg for nodes-uuid playbook

    Previously we tried setting from within the Ceph service some Ansible
    settings before launching the nodes-uuid playbook; we should instead
    just point to the global ansible.cfg file.

    Also includes setting environment variables not in the ansible.cfg.

    Co-Authored-By: John Fulton <email address hidden>
    Closes-Bug: #1782423
    Change-Id: Id220ebba5d9881eae943c7459b2328d3384b7370

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 9.0.0.0rc1 release candidate.