Undercloud update installs outdated puppet modules

Bug #1597438 reported by Steven Hardy on 2016-06-29
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Steven Hardy

Bug Description

I update my undercloud and re-ran openstack undercloud install, but it's been persistently hanging when trying to start nova-compute, with this error:

[root@instack ~]# tail -f /var/log/nova/nova-compute.log
2016-06-29 16:53:13.088 25494 ERROR oslo.messaging._drivers.impl_rabbit [req-4f04d1e6-281b-473c-ba8d-cefc3f5bebbf - - - - -] AMQP server on is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds.

It appears it's trying to connect to localhost, not br-ctlplane where rabbit is listening, and there's no connection details or credentials in /etc/nova/nova.conf.

Adding this in manually (copied from another service config file), then restarting first openstack-nova-conductor, then openstack-nova-compute works around the issue.

rabbit_hosts= # IP of br-ctlplane
rabbit_userid = <ID>
rabbit_password = <PASSWORD>

Observed with:

$ rpm -q puppet-nova
[stack@instack ~]$ rpm -q instack-undercloud

Changed in tripleo:
status: New → Triaged
milestone: none → newton-2
summary: - re-running uncercloud install fails with hanging nova-compute
+ re-running undercloud install fails with hanging nova-compute
Steven Hardy (shardy) wrote :

So I found a workaround, which appears to prove my theory that missing rabbit config is the problem.

I assume nova or puppet-nova changed something but I've not yet figured out what.

in my undercloud.conf I added:

  hieradata_override = /home/stack/puppet-stack-config-override.yaml


  nova::rabbit_userid: 4ced9bd8aa7ac3000a4f8e5cebebbe96a29096c1
  nova::rabbit_password: eec0f0b25fea54a6b819519a6fda80868948b9ee

Note that rabbit_host didn't work, nova-compute still ended up trying to connect to localhost (possibly a puppet-nova bug?) but rabbit_hosts works

I'll post a patch adding these to instack-undercloud as it seems like we need to pass them in the puppet-stack-config hieradata.

Changed in tripleo:
importance: Undecided → High
assignee: nobody → Steven Hardy (shardy)
Steven Hardy (shardy) wrote :

Oh, actually this looks like it may be related to:


which switches to default_transport_url, consumed in instack-undercloud here:


This doesn't appear to update the nova.conf correctly.

Steven Hardy (shardy) wrote :

Hmm, so actually this appears to be a repo problem - after you've installed your instack VM, it'll pull puppet-nova from the current-tripleo pin, which gives me a version prior to the transport_url change, but instack-undercloud is specified in the delorean-current includpkgs whitelist, so it expects this newer version.

Adding puppet-* to the delorean-current whitelist appears to work around the problem, but I guess this is something we should fix within our instack/tripleo.sh workflow, as this will work fine for CI but potentially break in non obvious ways for locally upgrading developers...

Changed in tripleo:
importance: High → Medium
milestone: newton-2 → newton-3
summary: - re-running undercloud install fails with hanging nova-compute
+ Undercloud update installs outdated puppet modules
Steven Hardy (shardy) wrote :

To clarify, the workaround is to edit /etc/yum.repos.d/delorean-current.repo so includepkgs looks like:


This will then pull latest master for all the puppet modules

Emilien Macchi (emilienm) wrote :

The bug looks like outdated, I could upgrade my undercloud to latest trunk.

Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers