Undercloud update installs outdated puppet modules

Bug #1597438 reported by Steven Hardy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Medium
Steven Hardy

Bug Description

I update my undercloud and re-ran openstack undercloud install, but it's been persistently hanging when trying to start nova-compute, with this error:

[root@instack ~]# tail -f /var/log/nova/nova-compute.log
2016-06-29 16:53:13.088 25494 ERROR oslo.messaging._drivers.impl_rabbit [req-4f04d1e6-281b-473c-ba8d-cefc3f5bebbf - - - - -] AMQP server on 127.0.0.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds.

It appears it's trying to connect to localhost, not br-ctlplane where rabbit is listening, and there's no connection details or credentials in /etc/nova/nova.conf.

Adding this in manually (copied from another service config file), then restarting first openstack-nova-conductor, then openstack-nova-compute works around the issue.

[oslo_messaging_rabbit]
rabbit_hosts=192.0.2.1 # IP of br-ctlplane
rabbit_userid = <ID>
rabbit_password = <PASSWORD>

Revision history for this message
Steven Hardy (shardy) wrote : Re: re-running undercloud install fails with hanging nova-compute

Observed with:

$ rpm -q puppet-nova
puppet-nova-9.0.0-0.20160611195743.7ed4409.el7.centos.noarch
[stack@instack ~]$ rpm -q instack-undercloud
instack-undercloud-5.0.0-0.20160628063533.b8c5ac7.el7.centos.noarch

Changed in tripleo:
status: New → Triaged
milestone: none → newton-2
summary: - re-running uncercloud install fails with hanging nova-compute
+ re-running undercloud install fails with hanging nova-compute
Revision history for this message
Steven Hardy (shardy) wrote :

So I found a workaround, which appears to prove my theory that missing rabbit config is the problem.

I assume nova or puppet-nova changed something but I've not yet figured out what.

in my undercloud.conf I added:

  hieradata_override = /home/stack/puppet-stack-config-override.yaml

Containing:

  nova::rabbit_userid: 4ced9bd8aa7ac3000a4f8e5cebebbe96a29096c1
  nova::rabbit_password: eec0f0b25fea54a6b819519a6fda80868948b9ee
  nova::rabbit_hosts: 192.0.2.1

Note that rabbit_host didn't work, nova-compute still ended up trying to connect to localhost (possibly a puppet-nova bug?) but rabbit_hosts works

I'll post a patch adding these to instack-undercloud as it seems like we need to pass them in the puppet-stack-config hieradata.

Changed in tripleo:
importance: Undecided → High
assignee: nobody → Steven Hardy (shardy)
Revision history for this message
Steven Hardy (shardy) wrote :

Oh, actually this looks like it may be related to:

https://github.com/openstack/puppet-nova/commit/cb3db6925280ad50d2d11ca8aa73e9615ae29095

which switches to default_transport_url, consumed in instack-undercloud here:

https://github.com/openstack/instack-undercloud/commit/8ad6a8c10fb3f438b9b1ae9485fe06bcf03c5227

This doesn't appear to update the nova.conf correctly.

Revision history for this message
Steven Hardy (shardy) wrote :

Hmm, so actually this appears to be a repo problem - after you've installed your instack VM, it'll pull puppet-nova from the current-tripleo pin, which gives me a version prior to the transport_url change, but instack-undercloud is specified in the delorean-current includpkgs whitelist, so it expects this newer version.

Adding puppet-* to the delorean-current whitelist appears to work around the problem, but I guess this is something we should fix within our instack/tripleo.sh workflow, as this will work fine for CI but potentially break in non obvious ways for locally upgrading developers...

Changed in tripleo:
importance: High → Medium
milestone: newton-2 → newton-3
summary: - re-running undercloud install fails with hanging nova-compute
+ Undercloud update installs outdated puppet modules
Revision history for this message
Steven Hardy (shardy) wrote :

To clarify, the workaround is to edit /etc/yum.repos.d/delorean-current.repo so includepkgs looks like:

includepkgs=diskimage-builder,instack,instack-undercloud,os-apply-config,os-cloud-config,os-collect-config,os-net-config,os-refresh-config,python-tripleoclient,openstack-tripleo-common,openstack-tripleo-heat-templates,openstack-tripleo-image-elements,openstack-tripleo,openstack-tripleo-puppet-elements,puppet-*

This will then pull latest master for all the puppet modules

Revision history for this message
Emilien Macchi (emilienm) wrote :

The bug looks like outdated, I could upgrade my undercloud to latest trunk.

Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.