Undercloud update installs outdated puppet modules

Bug #1597438 reported by Steven Hardy on 2016-06-29
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Medium
Steven Hardy

Bug Description

I update my undercloud and re-ran openstack undercloud install, but it's been persistently hanging when trying to start nova-compute, with this error:

[root@instack ~]# tail -f /var/log/nova/nova-compute.log
2016-06-29 16:53:13.088 25494 ERROR oslo.messaging._drivers.impl_rabbit [req-4f04d1e6-281b-473c-ba8d-cefc3f5bebbf - - - - -] AMQP server on 127.0.0.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds.

It appears it's trying to connect to localhost, not br-ctlplane where rabbit is listening, and there's no connection details or credentials in /etc/nova/nova.conf.

Adding this in manually (copied from another service config file), then restarting first openstack-nova-conductor, then openstack-nova-compute works around the issue.

[oslo_messaging_rabbit]
rabbit_hosts=192.0.2.1 # IP of br-ctlplane
rabbit_userid = <ID>
rabbit_password = <PASSWORD>

Observed with:

$ rpm -q puppet-nova
puppet-nova-9.0.0-0.20160611195743.7ed4409.el7.centos.noarch
[stack@instack ~]$ rpm -q instack-undercloud
instack-undercloud-5.0.0-0.20160628063533.b8c5ac7.el7.centos.noarch

Changed in tripleo:
status: New → Triaged
milestone: none → newton-2
summary: - re-running uncercloud install fails with hanging nova-compute
+ re-running undercloud install fails with hanging nova-compute
Steven Hardy (shardy) wrote :

So I found a workaround, which appears to prove my theory that missing rabbit config is the problem.

I assume nova or puppet-nova changed something but I've not yet figured out what.

in my undercloud.conf I added:

  hieradata_override = /home/stack/puppet-stack-config-override.yaml

Containing:

  nova::rabbit_userid: 4ced9bd8aa7ac3000a4f8e5cebebbe96a29096c1
  nova::rabbit_password: eec0f0b25fea54a6b819519a6fda80868948b9ee
  nova::rabbit_hosts: 192.0.2.1

Note that rabbit_host didn't work, nova-compute still ended up trying to connect to localhost (possibly a puppet-nova bug?) but rabbit_hosts works

I'll post a patch adding these to instack-undercloud as it seems like we need to pass them in the puppet-stack-config hieradata.

Changed in tripleo:
importance: Undecided → High
assignee: nobody → Steven Hardy (shardy)
Steven Hardy (shardy) wrote :

Oh, actually this looks like it may be related to:

https://github.com/openstack/puppet-nova/commit/cb3db6925280ad50d2d11ca8aa73e9615ae29095

which switches to default_transport_url, consumed in instack-undercloud here:

https://github.com/openstack/instack-undercloud/commit/8ad6a8c10fb3f438b9b1ae9485fe06bcf03c5227

This doesn't appear to update the nova.conf correctly.

Steven Hardy (shardy) wrote :

Hmm, so actually this appears to be a repo problem - after you've installed your instack VM, it'll pull puppet-nova from the current-tripleo pin, which gives me a version prior to the transport_url change, but instack-undercloud is specified in the delorean-current includpkgs whitelist, so it expects this newer version.

Adding puppet-* to the delorean-current whitelist appears to work around the problem, but I guess this is something we should fix within our instack/tripleo.sh workflow, as this will work fine for CI but potentially break in non obvious ways for locally upgrading developers...

Changed in tripleo:
importance: High → Medium
milestone: newton-2 → newton-3
summary: - re-running undercloud install fails with hanging nova-compute
+ Undercloud update installs outdated puppet modules
Steven Hardy (shardy) wrote :

To clarify, the workaround is to edit /etc/yum.repos.d/delorean-current.repo so includepkgs looks like:

includepkgs=diskimage-builder,instack,instack-undercloud,os-apply-config,os-cloud-config,os-collect-config,os-net-config,os-refresh-config,python-tripleoclient,openstack-tripleo-common,openstack-tripleo-heat-templates,openstack-tripleo-image-elements,openstack-tripleo,openstack-tripleo-puppet-elements,puppet-*

This will then pull latest master for all the puppet modules

Emilien Macchi (emilienm) wrote :

The bug looks like outdated, I could upgrade my undercloud to latest trunk.

Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers