[Rocky/Master] periodic-tripleo-centos-7-master-containers-build jobs are timing out at undercloud install

[chandan kumar]

On Promotion pipeline, periodic-tripleo-centos-7-master-containers-build are timing out during undercloud install step for Rocky and Master branch.

Below is the logs for different timedout jobs:
[master]
http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build/e338815/job-output.txt.gz

http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build/5662422/job-output.txt.gz

http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build/18c0276/job-output.txt.gz

[Rocky]
http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-rocky-containers-build/9c2f189/job-output.txt.gz

http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-rocky-containers-build/9e40685/job-output.txt.gz

http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-rocky-containers-build/beb5cc4/job-output.txt.gz

It is a tracker bug, I am still investigating.

[chandan kumar (chkumar246)]

Here is the passed log: http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build/a5e3452/logs/undercloud/home/zuul/undercloud_install.log.txt.gz
 in this one:
/lib/docker-config-scripts/nova_api_discover_hosts.sh:/nova_api_discover_hosts.sh:ro docker.io/tripleomaster/centos-binary-nova-api:3ed8ac0e93367a02ad53d9fa93467057724b6621_fd8eb74b /usr/bin/bootstrap_host_exec nova_api /nova_api_discover_hosts.sh",
2018-11-05 15:21:15 | "(cellv2) Running cell_v2 host discovery",
2018-11-05 15:21:15 | "(cellv2) Waiting 600 seconds for hosts to register",
2018-11-05 15:21:15 | "(cellv2) compute node upstream-centos-7-rdo-cloud-tripleo-0000285905.localdomain has registered",
2018-11-05 15:21:15 | "(cellv2) All nodes registered",
2018-11-05 15:21:15 | "(cellv2) Running host discovery...",
2018-11-05 15:21:15 | "Found 2 cell mappings.",
2018-11-05 15:21:15 | "Skipping cell0 since it does not contain hosts.",
2018-11-05 15:21:15 | "Getting computes from cell 'default': 11cb9d41-89f4-41d2-a4c6-7260c026fe7f",
2018-11-05 15:21:15 | "Creating host mapping for service upstream-centos-7-rdo-cloud-tripleo-0000285905.localdomain",
2018-11-05 15:21:15 | "Found 1 unmapped computes in cell: 11cb9d41-89f4-41d2-a4c6-7260c026fe7f",

And in the timedout one:
http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build/72fe1a4/logs/undercloud/home/zuul/undercloud_install.log.txt.gz

/lib/docker-config-scripts/nova_api_discover_hosts.sh:/nova_api_discover_hosts.sh:ro 192.168.24.1:8787/tripleomaster/centos-binary-nova-api:tripleo-ci-testing-updated-20181107225415 /usr/bin/bootstrap_host_exec nova_api /nova_api_discover_hosts.sh",
2018-11-07 23:34:45 | "(cellv2) Running cell_v2 host discovery",
2018-11-07 23:34:45 | "(cellv2) Waiting 600 seconds for hosts to register",
2018-11-07 23:34:45 | "(cellv2) compute node upstream-centos-7-rdo-cloud-tripleo-0000290997.localdomain has not registered",
2018-11-07 23:34:45 | "(cellv2) Waiting 593 seconds for hosts to register",
2018-11-07 23:34:45 | "(cellv2) Waiting 557 seconds for hosts to register",
2018-11-07 23:34:45 | "(cellv2) Waiting 520 seconds for hosts to register",
2018-11-07 23:34:45 | "(cellv2) Waiting 488 seconds for hosts to register",
2018-11-07 23:34:45 | "(cellv2) Waiting 451 seconds for hosts to register",
2018-11-07 23:34:45 | "(cellv2) Waiting 419 seconds for hosts to register",
2018-11-07 23:34:45 | "(cellv2) Waiting 387 seconds for hosts to register",
2018-11-07 23:34:45 | "(cellv2) Waiting 354 seconds for hosts to register",
2018-11-07 23:34:45 | "(cellv2) Waiting 322 seconds for hosts to register",
2018-11-07 23:34:45 | "(cellv2) Waiting 290 seconds for hosts to register",
2018-11-07 23:34:45 | "(cellv2) Waiting 257 seconds for hosts to register",
2018-11-07 23:34:45 | "(cellv2) Waiting 225 seconds for hosts to register",
2018-11-07 23:34:45...

[chandan kumar (chkumar246)]

Not Sure, it might be related to this review: https://review.openstack.org/#/c/615730/,

[chandan kumar (chkumar246)]

Based on discussion with jpena, https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build/9527eee/logs/ara_oooq/result/4d3d7cee-bede-423e-8ddd-b154b5a452c0/ might be causing the issue.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/617308

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/617514

[OpenStack Infra (hudson-openstack)]

Change abandoned by Chandan Kumar (<email address hidden>) on branch: master
Review: https://review.openstack.org/617514
Reason: not needed

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/617308
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=69122df55c016290d8b8f556053834bba41c469f
Submitter: Zuul
Branch: master

commit 69122df55c016290d8b8f556053834bba41c469f
Author: Chandan Kumar <email address hidden>
Date: Mon Nov 12 17:54:09 2018 +0530

    Set undercloud_cloud_domain/hostname in fs023

    * https://review.openstack.org/#/c/615730/ sets undercloud_hostname
      based on ansible_version which does not get overridden in /etc/
      hosts leading to not registering compute domain names and hence
      it causes timed out issue while deploying the undercloud.

    Change-Id: I3fa8c4015efed2e12d1349e230adfb7d64ec24c0
    Closes-Bug:#1802880

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-quickstart 2.1.1 release.