stein undercloud hangs on ** Performing Heat stack create.. **

[wes hayutin]

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build/8c1e595/logs/undercloud/home/zuul/install-undercloud.log.txt.gz

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build/d752cb3/logs/undercloud/home/zuul/install-undercloud.log.txt.gz

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build/0632bfe/logs/undercloud/home/zuul/install-undercloud.log.txt.gz

No trace in the heat process log
https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build/8c1e595/logs/undercloud/var/log/heat-launcher/undercloud_deploy-4zl4Ap/heat.log.txt.gz

2018-10-14 04:59:38.058 10289 WARNING oslo_config.cfg [-] Deprecated: Option "deferred_auth_method" from group "DEFAULT" is deprecated for removal (Stored password based deferred auth is broken when used with keystone v3 and is not supported.). Its value may be silently ignored in the future.
2018-10-14 04:59:38.066 10289 INFO heat.common.wsgi [-] Starting single process server
2018-10-14 04:59:38.066 10289 INFO eventlet.wsgi.server [-] (10289) wsgi starting up on http://127.0.0.1:8006
2018-10-14 04:59:38.095 10289 INFO eventlet.wsgi.server [req-ad796b02-ceb7-48e9-9ab2-c389d40d78df admin admin - - -] 127.0.0.1 - - [14/Oct/2018 04:59:38] "GET / HTTP/1.1" 300 324 0.005872
2018-10-14 04:59:48.362 10289 INFO heat.common.wsgi [req-7f14945f-e1a6-4558-8278-5de3f6087a2d admin admin - - -] Processing request: POST /v1/admin/stacks

Command in master invoked:
https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build/8c1e595/logs/undercloud/var/log/journal.txt.gz#_Oct_14_04_59_33

Cron related??
https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build/8c1e595/logs/undercloud/var/log/journal.txt.gz#_Oct_14_07_49_02

Heat is NOT running when logs are collected..
https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build/8c1e595/logs/undercloud/var/log/extra/pstree.txt.gz

[Cédric Jeanneret (cjeanner)]

I'm able to reproduce this for both "undercloud" and "standalone" deploys.

I have one running (well, stuck) right now, I can provide some tmate session if wanted/needed.

[Rabi Mishra (rabi)]

It seems https://github.com/openstack/oslo.messaging/commit/d3f6ca0b47e2a5a3413b27617d9655f02b191c6a has removed the rpc_backend and it's now defaults to rabbit. Probably the reason for this failure?

May be https://review.openstack.org/610512 would help.

[Cédric Jeanneret (cjeanner)]

Apparently linked to : https://review.openstack.org/#/c/608196/

[Cédric Jeanneret (cjeanner)]

duh, wrong link - https://review.openstack.org/#/c/580910/

[Dan Prince (dan-prince)]

Just had a nice sync with cjeaner and therve on a tmate. We determined that the rpc_backend is deprecated and we just need to switch over to set transport_url = fake:// in the heat.conf for the heat-all binary.

[Bogdan Dobrelya (bogdando)]

https://review.openstack.org/#/c/610512 should help indeed

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/610512
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=2d8dc588f85625a0ccbf366de9852377a966c311
Submitter: Zuul
Branch: master

commit 2d8dc588f85625a0ccbf366de9852377a966c311
Author: rabi <email address hidden>
Date: Mon Oct 15 18:45:37 2018 +0530

    Use transport_url instead of rpc_backend

    oslo.messaging has removed rpc_backend in commit
    d3f6ca0b47e2a5a3413b27617d9655f02b191c6a, use transport_url instead.

    Change-Id: Ic51c79c0c0a1338e7a8fab998ceb6413de18920a
    Closes-Bug: #1797781

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-tripleoclient 11.1.0 release.