Overcloud deploy error creating overcloudrc

[Quique Llorente]

Looking at some jobs that timedout we found that the were failing creating overcloudrc
http://logs.openstack.org/91/587491/4/check/tripleo-ci-centos-7-containers-multinode/f618803/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-09-25_08_20_00
2018-09-25 08:19:46 | "stdout: Running batches of 50 until complete",
2018-09-25 08:19:46 | "+---------------------------------------------+--------------+-----------+",
2018-09-25 08:19:46 | "| Migration | Total Needed | Completed |",
2018-09-25 08:19:46 | "| create_incomplete_consumers | 0 | 0 |",
2018-09-25 08:19:46 | "| delete_build_requests_with_no_instance_uuid | 0 | 0 |",
2018-09-25 08:19:46 | "| migrate_instances_add_request_spec | 0 | 0 |",
2018-09-25 08:20:00 | "| migrate_keypairs_to_api_db | 0 | 0 |Waiting for messages on queue 'tripleo' with no timeout.
2018-09-25 09:13:26 | Exception creating overcloudrc: Ansible errors file not found at /var/lib/mistral/overcloud/ansible-errors.json
2018-09-25 09:13:26 | END return value: 1
2018-09-25 09:13:26 | ",
2018-09-25 09:13:26 | "| migrate_quota_classes_to_api_db | 0 | 0 |",
2018-09-25 09:13:26 | "| migrate_quota_limits_to_api_db | 0 | 0 |",
2018-09-25 09:13:26 | "| migration_migrate_to_uuid | 0 | 0 |",
2018-09-25 09:13:26 | "| populate_missing_availability_zones | 0 | 0 |",
2018-09-25 09:13:26 | "| populate_queued_for_delete | 0 | 0 |",
2018-09-25 09:13:26 | "| populate_uuids | 0 | 0 |",
2018-09-25 09:13:26 | "| service_uuids_online_data_migration | 0 | 0 |"
2018-09-25 09:13:26 | ]
2018-09-25 09:13:26 | }
2018-09-25 09:13:26 |

We still don't know if it's related to timeouts

[yatin (yatinkarel)]

Some more logs which might help in diagnosis:-
http://logs.openstack.org/47/604447/2/check/tripleo-ci-centos-7-containers-multinode/e18e77f/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
started with --timeout 108, stack creation completed in 10 minutes, ansible deployment took around 30 minutes, overcloud rc action stuck for around 75 minutes[1].
[1] http://logs.openstack.org/47/604447/2/check/tripleo-ci-centos-7-containers-multinode/e18e77f/logs/undercloud/var/log/containers/mistral/executor.log.txt.gz#_2018-09-25_21_17_13_901

http://logs.openstack.org/13/605013/1/check/tripleo-ci-centos-7-containers-multinode/0a2bd1d/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
started with --timeout 86, stack creation completed in 17 minutes, ansible deployment took around 30 minutes, overcloud rc action stuck for around 53 minutes[2].
[2] http://logs.openstack.org/13/605013/1/check/tripleo-ci-centos-7-containers-multinode/0a2bd1d/logs/undercloud/var/log/containers/mistral/executor.log.1.gz

[Quique Llorente (quiquell)]

Could be related to https://review.openstack.org/#/c/603802/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/605377

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/605377
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=2a26ef2cf19571e39fc3071b19500808c6d14fcc
Submitter: Zuul
Branch: master

commit 2a26ef2cf19571e39fc3071b19500808c6d14fcc
Author: Thomas Herve <email address hidden>
Date: Wed Sep 26 11:32:08 2018 +0200

    Start websocket client before workflows

    When we start a workflow in the client, we need to create the websocket
    connection beforehand. If the workflow is very quick (like
    create_overcloudrc), it could be finished before we subscribe to the
    Zaqar queue properly, and thus we wouldn't get the message.

    Change-Id: I56c5f3a094094f7ba2158d8a434122ccb496f6b4
    Closes-Bug: #1794418

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/605499

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/605499
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=dae0133410348363e916d0a398a3132ebdd2688c
Submitter: Zuul
Branch: stable/rocky

commit dae0133410348363e916d0a398a3132ebdd2688c
Author: Thomas Herve <email address hidden>
Date: Wed Sep 26 11:32:08 2018 +0200

    Start websocket client before workflows

    When we start a workflow in the client, we need to create the websocket
    connection beforehand. If the workflow is very quick (like
    create_overcloudrc), it could be finished before we subscribe to the
    Zaqar queue properly, and thus we wouldn't get the message.

    Change-Id: I56c5f3a094094f7ba2158d8a434122ccb496f6b4
    Closes-Bug: #1794418
    (cherry picked from commit 2a26ef2cf19571e39fc3071b19500808c6d14fcc)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-tripleoclient 11.0.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-tripleoclient 10.6.1 release.