Major upgrades from Queens to Master/Rocky fail during "overcloud upgrade run" step when having two or more hosts defined under the "overcloud" ansible group:
u'PLAY [Common roles for TripleO servers] ****************************************',
u'',
u'TASK [tripleo-bootstrap : Deploy required packages to bootstrap TripleO] *******',
u'Wednesday 11 July 2018 19:08:49 +0000 (0:00:00.155) 0:00:05.423 ******** ',
u'ok: [overcloud-controller-0] => {"changed": false, "msg": "", "rc": 0, "results": ["openstack-heat-agents-1.5.4-0.20180308153305.ecf43c7.el7.centos.noarch providing openstack-heat-agents is already installed", "jq-1.5-10.el7.x86_64 providing jq is already installed"]}',
u'',
u'TASK [tripleo-bootstrap : Create /var/lib/heat-config/tripleo-config-download directory for deployment data] ***',
u'Wednesday 11 July 2018 19:08:50 +0000 (0:00:01.112) 0:00:06.535 ******** ',
u'ok: [overcloud-controller-0] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/heat-config/tripleo-config-download", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 6, "state": "directory", "uid": 0}',
u'',
u'TASK [tripleo-ssh-known-hosts : Template /etc/ssh/ssh_known_hosts] *************',
u'Wednesday 11 July 2018 19:08:50 +0000 (0:00:00.569) 0:00:07.105 ******** ',
u'fatal: [overcloud-controller-0]: FAILED! => {"changed": false, "msg": "AnsibleUndefinedVariable: \'dict object\' has no attribute \'ansible_hostname\'"}',
u'',
u'NO MORE HOSTS LEFT *************************************************************',
u'',
u'PLAY RECAP *********************************************************************',
u'overcloud-controller-0 : ok=4 changed=0 unreachable=0 failed=1 ',
u'',
u'Wednesday 11 July 2018 19:08:51 +0000 (0:00:00.328) 0:00:07.434 ******** ',
The cause of this issue is in the fact that the "overcloud upgrade run" is being run with the option --roles Controller. This option limits the playbooks execution to a single role, in this case the "Controller" one. Therefore, when gathering facts in [0], we are gathering only for the Overcloud's controller, leaving any other host defined as children of the group overcloud without facts gathering. Then, deploy_steps_playbook.yaml invokes the role tripleo-ssh-known-hosts, which creates the following template depending on the groups['overcloud'] var [1]. groups['overcloud'] var contains all the hosts in the overcloud (controller, compute, cepth, etc...) but the facts_gathering step was run only for the controller, so when accessing to other hosts ansible facts the task fails.
[0] - https://github.com/openstack/tripleo-heat-templates/blob/master/common/deploy-steps.j2#L405
[1] - https://github.com/openstack/tripleo-common/blob/master/roles/tripleo-ssh-known-hosts/templates/ssh_known_hosts.j2#L1
Fix proposed to branch: master
Review: https:/