Undercloud update installs outdated puppet modules

[Steven Hardy]

I update my undercloud and re-ran openstack undercloud install, but it's been persistently hanging when trying to start nova-compute, with this error:

[root@instack ~]# tail -f /var/log/nova/nova-compute.log
2016-06-29 16:53:13.088 25494 ERROR oslo.messaging._drivers.impl_rabbit [req-4f04d1e6-281b-473c-ba8d-cefc3f5bebbf - - - - -] AMQP server on 127.0.0.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds.

It appears it's trying to connect to localhost, not br-ctlplane where rabbit is listening, and there's no connection details or credentials in /etc/nova/nova.conf.

Adding this in manually (copied from another service config file), then restarting first openstack-nova-conductor, then openstack-nova-compute works around the issue.

[oslo_messaging_rabbit]
rabbit_hosts=192.0.2.1 # IP of br-ctlplane
rabbit_userid = <ID>
rabbit_password = <PASSWORD>

[Steven Hardy (shardy)]

Observed with:

$ rpm -q puppet-nova
puppet-nova-9.0.0-0.20160611195743.7ed4409.el7.centos.noarch
[stack@instack ~]$ rpm -q instack-undercloud
instack-undercloud-5.0.0-0.20160628063533.b8c5ac7.el7.centos.noarch

[Steven Hardy (shardy)]

So I found a workaround, which appears to prove my theory that missing rabbit config is the problem.

I assume nova or puppet-nova changed something but I've not yet figured out what.

in my undercloud.conf I added:

  hieradata_override = /home/stack/puppet-stack-config-override.yaml

Containing:

  nova::rabbit_userid: 4ced9bd8aa7ac3000a4f8e5cebebbe96a29096c1
  nova::rabbit_password: eec0f0b25fea54a6b819519a6fda80868948b9ee
  nova::rabbit_hosts: 192.0.2.1

Note that rabbit_host didn't work, nova-compute still ended up trying to connect to localhost (possibly a puppet-nova bug?) but rabbit_hosts works

I'll post a patch adding these to instack-undercloud as it seems like we need to pass them in the puppet-stack-config hieradata.

[Steven Hardy (shardy)]

Oh, actually this looks like it may be related to:

https://github.com/openstack/puppet-nova/commit/cb3db6925280ad50d2d11ca8aa73e9615ae29095

which switches to default_transport_url, consumed in instack-undercloud here:

https://github.com/openstack/instack-undercloud/commit/8ad6a8c10fb3f438b9b1ae9485fe06bcf03c5227

This doesn't appear to update the nova.conf correctly.

[Steven Hardy (shardy)]

Hmm, so actually this appears to be a repo problem - after you've installed your instack VM, it'll pull puppet-nova from the current-tripleo pin, which gives me a version prior to the transport_url change, but instack-undercloud is specified in the delorean-current includpkgs whitelist, so it expects this newer version.

Adding puppet-* to the delorean-current whitelist appears to work around the problem, but I guess this is something we should fix within our instack/tripleo.sh workflow, as this will work fine for CI but potentially break in non obvious ways for locally upgrading developers...

[Steven Hardy (shardy)]

To clarify, the workaround is to edit /etc/yum.repos.d/delorean-current.repo so includepkgs looks like:

includepkgs=diskimage-builder,instack,instack-undercloud,os-apply-config,os-cloud-config,os-collect-config,os-net-config,os-refresh-config,python-tripleoclient,openstack-tripleo-common,openstack-tripleo-heat-templates,openstack-tripleo-image-elements,openstack-tripleo,openstack-tripleo-puppet-elements,puppet-*

This will then pull latest master for all the puppet modules

[Emilien Macchi (emilienm)]

The bug looks like outdated, I could upgrade my undercloud to latest trunk.