Ocata -> Pike overcloud upgrade: nova_compute is unable to start post upgrade

Bug #1699315 reported by Marius Cornea
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Expired
Undecided
Unassigned

Bug Description

Steps to reproduce:

1. Install Ocata undercloud

2. Deploy Ocata overcloud with one controller node (nova compute services are also running on it):
source ~/stackrc
export THT=/usr/share/openstack-tripleo-heat-templates/

openstack overcloud deploy --templates $THT \
-r ~/openstack_deployment/roles/roles_data.yaml \
-e $THT/environments/network-isolation.yaml \
-e $THT/environments/network-management.yaml \
-e $THT/environments/tls-endpoints-public-ip.yaml \
-e ~/openstack_deployment/environments/nodes.yaml \
-e ~/openstack_deployment/environments/network-environment.yaml \
-e ~/openstack_deployment/environments/public_vip.yaml \
-e ~/openstack_deployment/environments/enable-tls.yaml \
-e ~/openstack_deployment/environments/inject-trust-anchor.yaml \
-e ~/openstack_deployment/environments/neutron-settings.yaml

roles_data.yaml:
http://paste.openstack.org/show/613197/

3. Upgrade undercloud to Pike

4. Download container images:
openstack overcloud container image upload --verbose --config-file /usr/share/tripleo-common/container-images/overcloud_containers.yaml

5. Replace repos on overcloud node to Pike repos:

sudo mv /etc/yum.repos.d/delorean* repo/
sudo curl -L -o /etc/yum.repos.d/delorean.repo https://trunk.rdoproject.org/centos7-master/current-passed-ci/delorean.repo
sudo curl -L -o /etc/yum.repos.d/delorean-current.repo https://trunk.rdoproject.org/centos7/current/delorean.repo
sudo sed -i 's/\[delorean\]/\[delorean-current\]/' /etc/yum.repos.d/delorean-current.repo
sudo /bin/bash -c "cat <<EOF>>/etc/yum.repos.d/delorean-current.repo
includepkgs=diskimage-builder,instack,instack-undercloud,os-apply-config,os-collect-config,os-net-config,os-refresh-config,python-tripleoclient,openstack-tripleo-common*,openstack-tripleo-heat-templates,openstack-tripleo-image-elements,openstack-tripleo,openstack-tripleo-puppet-elements,openstack-puppet-modules,openstack-tripleo-ui,puppet-*
EOF"
sudo curl -L -o /etc/yum.repos.d/delorean-deps.repo https://trunk.rdoproject.org/centos7/delorean-deps.repo

6. # create an env file to make overcloud fetch the images from the undercloud
# (192.168.0.1 is undercloud IP that must be pingable from the overcloud)
echo > ~/containers-default-parameters.yaml 'parameter_defaults:
  DockerNamespace: 192.168.0.1:8787/tripleoupstream
  DockerNamespaceIsRegistry: true
'

7. Upgrade overcloud:

source ~/stackrc
export THT=/usr/share/openstack-tripleo-heat-templates/

openstack overcloud deploy --templates $THT \
-r ~/openstack_deployment/roles/roles_data.yaml \
-e $THT/environments/network-isolation.yaml \
-e $THT/environments/network-management.yaml \
-e $THT/environments/tls-endpoints-public-ip.yaml \
-e ~/openstack_deployment/environments/nodes.yaml \
-e ~/openstack_deployment/environments/network-environment.yaml \
-e ~/openstack_deployment/environments/public_vip.yaml \
-e ~/openstack_deployment/environments/enable-tls.yaml \
-e ~/openstack_deployment/environments/inject-trust-anchor.yaml \
-e ~/openstack_deployment/environments/neutron-settings.yaml \
-e ~/containers-default-parameters.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-composable-steps-docker.yaml

Result:

http://paste.openstack.org/show/613211/

In /var/log/containers/nova/nova-compute.log we can notice:

2017-06-20 19:28:55.712 6 INFO nova.service [-] Starting compute node (version 16.0.0-0.20170616124201.8238e8a.el7.centos)
2017-06-20 19:28:55.749 6 ERROR nova.virt.libvirt.host [-] Connection to libvirt failed: authentication failed: access denied by policy: libvirtError: authentication failed: access denied by policy
2017-06-20 19:28:55.749 6 ERROR nova.virt.libvirt.host Traceback (most recent call last):
2017-06-20 19:28:55.749 6 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 454, in get_connection
2017-06-20 19:28:55.749 6 ERROR nova.virt.libvirt.host conn = self._get_connection()
2017-06-20 19:28:55.749 6 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 437, in _get_connection
2017-06-20 19:28:55.749 6 ERROR nova.virt.libvirt.host {'msg': ex})
2017-06-20 19:28:55.749 6 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-06-20 19:28:55.749 6 ERROR nova.virt.libvirt.host self.force_reraise()
2017-06-20 19:28:55.749 6 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-06-20 19:28:55.749 6 ERROR nova.virt.libvirt.host six.reraise(self.type_, self.value, self.tb)
2017-06-20 19:28:55.749 6 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 426, in _get_connection
2017-06-20 19:28:55.749 6 ERROR nova.virt.libvirt.host self._wrapped_conn = self._get_new_connection()
2017-06-20 19:28:55.749 6 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 371, in _get_new_connection
2017-06-20 19:28:55.749 6 ERROR nova.virt.libvirt.host wrapped_conn = self._connect(self._uri, self._read_only)
2017-06-20 19:28:55.749 6 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 227, in _connect
2017-06-20 19:28:55.749 6 ERROR nova.virt.libvirt.host libvirt.openAuth, uri, auth, flags)
2017-06-20 19:28:55.749 6 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2017-06-20 19:28:55.749 6 ERROR nova.virt.libvirt.host rv = execute(f, *args, **kwargs)
2017-06-20 19:28:55.749 6 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2017-06-20 19:28:55.749 6 ERROR nova.virt.libvirt.host six.reraise(c, e, tb)
2017-06-20 19:28:55.749 6 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2017-06-20 19:28:55.749 6 ERROR nova.virt.libvirt.host rv = meth(*args, **kwargs)
2017-06-20 19:28:55.749 6 ERROR nova.virt.libvirt.host File "/usr/lib64/python2.7/site-packages/libvirt.py", line 105, in openAuth
2017-06-20 19:28:55.749 6 ERROR nova.virt.libvirt.host if ret is None:raise libvirtError('virConnectOpenAuth() failed')
2017-06-20 19:28:55.749 6 ERROR nova.virt.libvirt.host libvirtError: authentication failed: access denied by policy
2017-06-20 19:28:55.749 6 ERROR nova.virt.libvirt.host
2017-06-20 19:28:55.769 6 INFO nova.virt.libvirt.driver [-] Connection event '0' reason 'Failed to connect to libvirt: authentication failed: access denied by policy'
2017-06-20 19:28:55.826 6 ERROR oslo_service.service [req-484a1a42-5978-4e07-8a9c-c61ec8448dab - - - - -] Error starting thread.: HypervisorUnavailable: Connection to the hypervisor is broken on host: overcloud-controller-0.localdomain
2017-06-20 19:28:55.826 6 ERROR oslo_service.service Traceback (most recent call last):
2017-06-20 19:28:55.826 6 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 721, in run_service
2017-06-20 19:28:55.826 6 ERROR oslo_service.service service.start()
2017-06-20 19:28:55.826 6 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/service.py", line 143, in start
2017-06-20 19:28:55.826 6 ERROR oslo_service.service self.manager.init_host()
2017-06-20 19:28:55.826 6 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1125, in init_host
2017-06-20 19:28:55.826 6 ERROR oslo_service.service self.driver.init_host(host=self.host)
2017-06-20 19:28:55.826 6 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 457, in init_host
2017-06-20 19:28:55.826 6 ERROR oslo_service.service self._do_quality_warnings()
2017-06-20 19:28:55.826 6 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 434, in _do_quality_warnings
2017-06-20 19:28:55.826 6 ERROR oslo_service.service caps = self._host.get_capabilities()
2017-06-20 19:28:55.826 6 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 639, in get_capabilities
2017-06-20 19:28:55.826 6 ERROR oslo_service.service xmlstr = self.get_connection().getCapabilities()
2017-06-20 19:28:55.826 6 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 463, in get_connection
2017-06-20 19:28:55.826 6 ERROR oslo_service.service raise exception.HypervisorUnavailable(host=CONF.host)
2017-06-20 19:28:55.826 6 ERROR oslo_service.service HypervisorUnavailable: Connection to the hypervisor is broken on host: overcloud-controller-0.localdomain
2017-06-20 19:28:55.826 6 ERROR oslo_service.service
(END)

Tags: upgrade
Oliver Walsh (owalsh)
Changed in tripleo:
assignee: nobody → Oliver Walsh (owalsh)
Revision history for this message
Emilien Macchi (emilienm) wrote :

There are no currently open reviews on this bug, changing the status back to the previous state and unassigning. If there are active reviews related to this bug, please include links in comments.

Changed in tripleo:
assignee: Oliver Walsh (owalsh) → nobody
Changed in tripleo:
milestone: pike-3 → pike-rc1
Changed in tripleo:
milestone: pike-rc1 → pike-rc2
Changed in tripleo:
milestone: pike-rc2 → queens-1
Changed in tripleo:
milestone: queens-1 → queens-2
Changed in tripleo:
milestone: queens-2 → queens-3
Changed in tripleo:
milestone: queens-3 → queens-rc1
Changed in tripleo:
milestone: queens-rc1 → rocky-1
Changed in tripleo:
milestone: rocky-1 → rocky-2
Changed in tripleo:
milestone: rocky-2 → rocky-3
Changed in tripleo:
milestone: rocky-3 → rocky-rc1
Changed in tripleo:
milestone: rocky-rc1 → stein-1
Changed in tripleo:
milestone: stein-1 → stein-2
Revision history for this message
Emilien Macchi (emilienm) wrote : Cleanup EOL bug report

This is an automated cleanup. This bug report has been closed because it
is older than 18 months and there is no open code change to fix this.
After this time it is unlikely that the circumstances which lead to
the observed issue can be reproduced.

If you can reproduce the bug, please:
* reopen the bug report (set to status "New")
* AND add the detailed steps to reproduce the issue (if applicable)
* AND leave a comment "CONFIRMED FOR: <RELEASE_NAME>"
  Only still supported release names are valid (FUTURE, PIKE, QUEENS, ROCKY, STEIN).
  Valid example: CONFIRMED FOR: FUTURE

Changed in tripleo:
importance: High → Undecided
status: Triaged → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.