tripleo-ci-centos-7-3nodes-multinode timing out frequently in the gate

[Alex Schultz]

According to cistatus, the 3node job has timed out 6 times in the gate today 2018-04-30.

http://logs.openstack.org/80/564380/2/gate/tripleo-ci-centos-7-3nodes-multinode/e9352aa/

http://logs.openstack.org/78/560578/28/gate/tripleo-ci-centos-7-3nodes-multinode/3d8208d/

http://logs.openstack.org/71/564671/8/gate/tripleo-ci-centos-7-3nodes-multinode/3290300/

http://logs.openstack.org/19/563819/1/gate/tripleo-ci-centos-7-3nodes-multinode/1a38777/

http://logs.openstack.org/22/563322/1/gate/tripleo-ci-centos-7-3nodes-multinode/f6782b7/

http://logs.openstack.org/58/564758/2/gate/tripleo-ci-centos-7-3nodes-multinode/842ce56/

This may be similar to Bug 1767252 where we're seeing an increase in run times since the config-download switch.

[Alex Schultz (alex-schultz)]

See additional comments in Bug 1772403

[Alex Schultz (alex-schultz)]

It's failing because the controller services are not being correctly deployed on the 'controller' node. The only thing running on subnode-1 (controller) is mariadb, rabbitmq, memcache, haproxy and cinder-volume. The deployment is successful but the post deploy check in quickstart fails because the endpoint list is empty.

[Alex Schultz (alex-schultz)]

Actually in review of the roles file[0], the controller node is correctly deployed. We're missing something else on the ControlereApi node.

[0] https://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/multinode-3nodes.yaml#L59-L85

[Alex Schultz (alex-schultz)]

It appears it's broken because we don't actually run the boostrap bits for keystone because the ControllerApi node thinks that the Controller node is the bootstrap node for that role. It's not.

In the deployment log we see the 'Write docker-puppet-task json files' is skipped on the ControllerApi node.

http://logs.openstack.org/50/569550/8/check/tripleo-ci-centos-7-3nodes-multinode/fddbbcb/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-05-21_16_43_31

2018-05-21 16:43:31 | TASK [Write docker-puppet-tasks json files] ************************************
2018-05-21 16:43:31 | task path: /var/lib/mistral/1b504e8a-df7f-43e7-9673-ceaae3c65ffe/common_deploy_steps_tasks.yaml:160
2018-05-21 16:43:31 | skipping: [centos-7-rax-dfw-0004113538] => (item={'value': [{'puppet_tags': u'keystone_config,keystone_domain_config,keystone_endpoint,keystone_identity_provider,keystone_paste_ini,keystone_role,keystone_service,keystone_tenant,keystone_user,keystone_user_role,keystone_domain', 'config_volume': u'keystone_init_tasks', 'step_config': u'include ::tripleo::profile::base::keystone', 'config_image': u'192.168.24.1:8787/tripleomaster/centos-binary-keystone:20b99f6998c088650b0c0cb066cc6aac3e5f9312_b333f915'}], 'key': u'step_3'}) => {"changed": false, "item": {"key": "step_3", "value": [{"config_image": "192.168.24.1:8787/tripleomaster/centos-binary-keystone:20b99f6998c088650b0c0cb066cc6aac3e5f9312_b333f915", "config_volume": "keystone_init_tasks", "puppet_tags": "keystone_config,keystone_domain_config,keystone_endpoint,keystone_identity_provider,keystone_paste_ini,keystone_role,keystone_service,keystone_tenant,keystone_user,keystone_user_role,keystone_domain", "step_config": "include ::tripleo::profile::base::keystone"}]}, "skip_reason": "Conditional result was False"}

From common_deploy_steps_tasks.yaml the conditional is...

http://logs.openstack.org/50/569550/8/check/tripleo-ci-centos-7-3nodes-multinode/fddbbcb/logs/undercloud/var/lib/mistral/1b504e8a-df7f-43e7-9673-ceaae3c65ffe/common_deploy_steps_tasks.yaml.txt.gz

        - name: Write docker-puppet-tasks json files
          copy: content="{{item.value|to_json}}" dest=/var/lib/docker-puppet/docker-puppet-tasks{{item.key.replace("step_", "")}}.json force=yes mode=0600
          with_dict: "{{role_data_docker_puppet_tasks}}"
          when: deploy_server_id == bootstrap_server_id

If we look at the tripleo-ansible-inventory.yaml we see...

http://logs.openstack.org/50/569550/8/check/tripleo-ci-centos-7-3nodes-multinode/fddbbcb/logs/undercloud/var/lib/mistral/1b504e8a-df7f-43e7-9673-ceaae3c65ffe/tripleo-ansible-inventory.yaml.txt.gz

Controller:
  hosts:
    centos-7-rax-dfw-0004113541:
      <snip>
  vars: {ansible_ssh_user: tripleo-admin, bootstrap_server_id: ece919cf-6814-442d-89fa-0879678fcc7e,
    tripleo_role_name: Controller}
ControllerApi:
  hosts:
    centos-7-rax-dfw-0004113538:
      <snip>
  vars: {ansible_ssh_user: tripleo-admin, bootstrap_server_id: ece919cf-6814-442d-89fa-0879678fcc7e,
    tripleo_role_name: ControllerApi}

So because the bootstrap_server_id for the ControlerApi role is set to a Controller node, it never runs some of the steps which in this case includes t...

[Alex Schultz (alex-schultz)]

It looks like the bootstrap_server_id is determined here: https://github.com/openstack/tripleo-heat-templates/blob/1eaa019a1f1fbf126c8d9cbffdb608a02d96f3f7/overcloud.j2.yaml#L605-L618

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/570252

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/570275

[Bogdan Dobrelya (bogdando)]

As it was commented in IRC, a bootstrap calculation per service is the way to go. And we have it already implemented via https://github.com/openstack/tripleo-heat-templates/blob/7b74e8b90e04a7390598f70f8a7e00eabc7ec870/overcloud.j2.yaml#L724 , which seems missed adoption in puppets

[Bogdan Dobrelya (bogdando)]

An example of a partial adoption done for mysql puppets http://codesearch.openstack.org/?q=mysql_short_bootstrap_node_name&i=nope&files=&repos=

No need to backport this also, as it was introduced in Rocky, since we switched to config-download. In queens we had a bootstrap per-role.

Thanks @jaosorior and @shardy for the inputs!

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alex Schultz (<email address hidden>) on branch: master
Review: https://review.openstack.org/570275
Reason: it is passing now w/o this.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/570252
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=0ef3058459c082b0f19b4fe275c38cd8e60fb8d8
Submitter: Zuul
Branch: master

commit 0ef3058459c082b0f19b4fe275c38cd8e60fb8d8
Author: Alex Schultz <email address hidden>
Date: Wed May 23 13:12:54 2018 -0600

    Fix 3node deployment

    The keystone role needs to be on the same node as the mysql/haproxy node
    due to a TLS requirement that the haproxy be on the 'primary' tagged
    node. Since the keystone bootstrap stuff only runs on the primary node
    as well, they need to be tied together.

    Change-Id: Ifa7ed93993082466a2a6ddff56bee58b074be512
    Closes-Bug: #1768142

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 9.0.0.0b3 development milestone.