Comment 4 for bug 1768142

It appears it's broken because we don't actually run the boostrap bits for keystone because the ControllerApi node thinks that the Controller node is the bootstrap node for that role. It's not.

In the deployment log we see the 'Write docker-puppet-task json files' is skipped on the ControllerApi node.

http://logs.openstack.org/50/569550/8/check/tripleo-ci-centos-7-3nodes-multinode/fddbbcb/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-05-21_16_43_31

2018-05-21 16:43:31 | TASK [Write docker-puppet-tasks json files] ************************************
2018-05-21 16:43:31 | task path: /var/lib/mistral/1b504e8a-df7f-43e7-9673-ceaae3c65ffe/common_deploy_steps_tasks.yaml:160
2018-05-21 16:43:31 | skipping: [centos-7-rax-dfw-0004113538] => (item={'value': [{'puppet_tags': u'keystone_config,keystone_domain_config,keystone_endpoint,keystone_identity_provider,keystone_paste_ini,keystone_role,keystone_service,keystone_tenant,keystone_user,keystone_user_role,keystone_domain', 'config_volume': u'keystone_init_tasks', 'step_config': u'include ::tripleo::profile::base::keystone', 'config_image': u'192.168.24.1:8787/tripleomaster/centos-binary-keystone:20b99f6998c088650b0c0cb066cc6aac3e5f9312_b333f915'}], 'key': u'step_3'}) => {"changed": false, "item": {"key": "step_3", "value": [{"config_image": "192.168.24.1:8787/tripleomaster/centos-binary-keystone:20b99f6998c088650b0c0cb066cc6aac3e5f9312_b333f915", "config_volume": "keystone_init_tasks", "puppet_tags": "keystone_config,keystone_domain_config,keystone_endpoint,keystone_identity_provider,keystone_paste_ini,keystone_role,keystone_service,keystone_tenant,keystone_user,keystone_user_role,keystone_domain", "step_config": "include ::tripleo::profile::base::keystone"}]}, "skip_reason": "Conditional result was False"}

From common_deploy_steps_tasks.yaml the conditional is...

http://logs.openstack.org/50/569550/8/check/tripleo-ci-centos-7-3nodes-multinode/fddbbcb/logs/undercloud/var/lib/mistral/1b504e8a-df7f-43e7-9673-ceaae3c65ffe/common_deploy_steps_tasks.yaml.txt.gz

        - name: Write docker-puppet-tasks json files
          copy: content="{{item.value|to_json}}" dest=/var/lib/docker-puppet/docker-puppet-tasks{{item.key.replace("step_", "")}}.json force=yes mode=0600
          with_dict: "{{role_data_docker_puppet_tasks}}"
          when: deploy_server_id == bootstrap_server_id

If we look at the tripleo-ansible-inventory.yaml we see...

http://logs.openstack.org/50/569550/8/check/tripleo-ci-centos-7-3nodes-multinode/fddbbcb/logs/undercloud/var/lib/mistral/1b504e8a-df7f-43e7-9673-ceaae3c65ffe/tripleo-ansible-inventory.yaml.txt.gz

Controller:
  hosts:
    centos-7-rax-dfw-0004113541:
      <snip>
  vars: {ansible_ssh_user: tripleo-admin, bootstrap_server_id: ece919cf-6814-442d-89fa-0879678fcc7e,
    tripleo_role_name: Controller}
ControllerApi:
  hosts:
    centos-7-rax-dfw-0004113538:
      <snip>
  vars: {ansible_ssh_user: tripleo-admin, bootstrap_server_id: ece919cf-6814-442d-89fa-0879678fcc7e,
    tripleo_role_name: ControllerApi}

So because the bootstrap_server_id for the ControlerApi role is set to a Controller node, it never runs some of the steps which in this case includes the keystone_endpoints and more.