nova-compute cannot restart if _init_instance failed

Bug #1438680 reported by Pavel Kholkin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Fix Released
High
Pavel Kholkin
5.1.x
Fix Released
High
Alex Ermolov
6.0.x
Fix Released
High
Alex Ermolov
6.1.x
Fix Released
High
Pavel Kholkin

Bug Description

Revision history for this message
OSCI Robot (oscirobot) wrote :

RPM package nova has been built for project openstack/nova
Package version == 2014.2.2, package release == fuel6.0.1.mira9.git.d930a43.1009d65

Changeset: https://review.fuel-infra.org/5116
project: openstack/nova
branch: openstack-ci/fuel-6.0.1/2014.2
author: Kholkin Pavel
committer: Kholkin Pavel
subject: Compute: Catch binding failed exception while init host
status: patchset-created

Files placed on repository:
openstack-nova-2014.2.2-fuel6.0.1.mira9.git.d930a43.1009d65.noarch.rpm
openstack-nova-api-2014.2.2-fuel6.0.1.mira9.git.d930a43.1009d65.noarch.rpm
openstack-nova-cells-2014.2.2-fuel6.0.1.mira9.git.d930a43.1009d65.noarch.rpm
openstack-nova-cert-2014.2.2-fuel6.0.1.mira9.git.d930a43.1009d65.noarch.rpm
openstack-nova-common-2014.2.2-fuel6.0.1.mira9.git.d930a43.1009d65.noarch.rpm
openstack-nova-compute-2014.2.2-fuel6.0.1.mira9.git.d930a43.1009d65.noarch.rpm
openstack-nova-conductor-2014.2.2-fuel6.0.1.mira9.git.d930a43.1009d65.noarch.rpm
openstack-nova-console-2014.2.2-fuel6.0.1.mira9.git.d930a43.1009d65.noarch.rpm
openstack-nova-doc-2014.2.2-fuel6.0.1.mira9.git.d930a43.1009d65.noarch.rpm
openstack-nova-network-2014.2.2-fuel6.0.1.mira9.git.d930a43.1009d65.noarch.rpm
openstack-nova-novncproxy-2014.2.2-fuel6.0.1.mira9.git.d930a43.1009d65.noarch.rpm
openstack-nova-objectstore-2014.2.2-fuel6.0.1.mira9.git.d930a43.1009d65.noarch.rpm
openstack-nova-scheduler-2014.2.2-fuel6.0.1.mira9.git.d930a43.1009d65.noarch.rpm
python-nova-2014.2.2-fuel6.0.1.mira9.git.d930a43.1009d65.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0.1-stable-5116/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package nova has been built for project openstack/nova
Package version == 2014.2.2, package release == fuel6.0.1~mira9+git.d930a43.1009d65

Changeset: https://review.fuel-infra.org/5116
project: openstack/nova
branch: openstack-ci/fuel-6.0.1/2014.2
author: Kholkin Pavel
committer: Kholkin Pavel
subject: Compute: Catch binding failed exception while init host
status: patchset-created

Files placed on repository:
nova-ajax-console-proxy_2014.2.2-fuel6.0.1~mira9+git.d930a43.1009d65_all.deb
nova-api-ec2_2014.2.2-fuel6.0.1~mira9+git.d930a43.1009d65_all.deb
nova-api-metadata_2014.2.2-fuel6.0.1~mira9+git.d930a43.1009d65_all.deb
nova-api-os-compute_2014.2.2-fuel6.0.1~mira9+git.d930a43.1009d65_all.deb
nova-api-os-volume_2014.2.2-fuel6.0.1~mira9+git.d930a43.1009d65_all.deb
nova-api_2014.2.2-fuel6.0.1~mira9+git.d930a43.1009d65_all.deb
nova-baremetal_2014.2.2-fuel6.0.1~mira9+git.d930a43.1009d65_all.deb
nova-cells_2014.2.2-fuel6.0.1~mira9+git.d930a43.1009d65_all.deb
nova-cert_2014.2.2-fuel6.0.1~mira9+git.d930a43.1009d65_all.deb
nova-common_2014.2.2-fuel6.0.1~mira9+git.d930a43.1009d65_all.deb
nova-compute-kvm_2014.2.2-fuel6.0.1~mira9+git.d930a43.1009d65_all.deb
nova-compute-libvirt_2014.2.2-fuel6.0.1~mira9+git.d930a43.1009d65_all.deb
nova-compute-lxc_2014.2.2-fuel6.0.1~mira9+git.d930a43.1009d65_all.deb
nova-compute-qemu_2014.2.2-fuel6.0.1~mira9+git.d930a43.1009d65_all.deb
nova-compute-vmware_2014.2.2-fuel6.0.1~mira9+git.d930a43.1009d65_all.deb
nova-compute-xen_2014.2.2-fuel6.0.1~mira9+git.d930a43.1009d65_all.deb
nova-compute_2014.2.2-fuel6.0.1~mira9+git.d930a43.1009d65_all.deb
nova-conductor_2014.2.2-fuel6.0.1~mira9+git.d930a43.1009d65_all.deb
nova-console_2014.2.2-fuel6.0.1~mira9+git.d930a43.1009d65_all.deb
nova-consoleauth_2014.2.2-fuel6.0.1~mira9+git.d930a43.1009d65_all.deb
nova-doc_2014.2.2-fuel6.0.1~mira9+git.d930a43.1009d65_all.deb
nova-network_2014.2.2-fuel6.0.1~mira9+git.d930a43.1009d65_all.deb
nova-novncproxy_2014.2.2-fuel6.0.1~mira9+git.d930a43.1009d65_all.deb
nova-objectstore_2014.2.2-fuel6.0.1~mira9+git.d930a43.1009d65_all.deb
nova-scheduler_2014.2.2-fuel6.0.1~mira9+git.d930a43.1009d65_all.deb
nova-spiceproxy_2014.2.2-fuel6.0.1~mira9+git.d930a43.1009d65_all.deb
nova-volume_2014.2.2-fuel6.0.1~mira9+git.d930a43.1009d65_all.deb
nova-xvpvncproxy_2014.2.2-fuel6.0.1~mira9+git.d930a43.1009d65_all.deb
python-nova_2014.2.2-fuel6.0.1~mira9+git.d930a43.1009d65_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0.1-stable-5116/ubuntu

description: updated
tags: added: nova
Revision history for this message
OSCI Robot (oscirobot) wrote :

RPM package nova has been built for project openstack/nova
Package version == 2014.1.3, package release == fuel5.1.1.mira7.git.ed5d120.7ad20ad

Changeset: https://review.fuel-infra.org/5174
project: openstack/nova
branch: openstack-ci/fuel-5.1.1-updates/2014.1.1
author: Roman Podoliaka
committer: Roman Podoliaka
subject: Compute: Catch binding failed exception while init host
status: patchset-created

Files placed on repository:
openstack-nova-2014.1.3-fuel5.1.1.mira7.git.ed5d120.7ad20ad.noarch.rpm
openstack-nova-api-2014.1.3-fuel5.1.1.mira7.git.ed5d120.7ad20ad.noarch.rpm
openstack-nova-cells-2014.1.3-fuel5.1.1.mira7.git.ed5d120.7ad20ad.noarch.rpm
openstack-nova-cert-2014.1.3-fuel5.1.1.mira7.git.ed5d120.7ad20ad.noarch.rpm
openstack-nova-common-2014.1.3-fuel5.1.1.mira7.git.ed5d120.7ad20ad.noarch.rpm
openstack-nova-compute-2014.1.3-fuel5.1.1.mira7.git.ed5d120.7ad20ad.noarch.rpm
openstack-nova-conductor-2014.1.3-fuel5.1.1.mira7.git.ed5d120.7ad20ad.noarch.rpm
openstack-nova-console-2014.1.3-fuel5.1.1.mira7.git.ed5d120.7ad20ad.noarch.rpm
openstack-nova-doc-2014.1.3-fuel5.1.1.mira7.git.ed5d120.7ad20ad.noarch.rpm
openstack-nova-network-2014.1.3-fuel5.1.1.mira7.git.ed5d120.7ad20ad.noarch.rpm
openstack-nova-novncproxy-2014.1.3-fuel5.1.1.mira7.git.ed5d120.7ad20ad.noarch.rpm
openstack-nova-objectstore-2014.1.3-fuel5.1.1.mira7.git.ed5d120.7ad20ad.noarch.rpm
openstack-nova-scheduler-2014.1.3-fuel5.1.1.mira7.git.ed5d120.7ad20ad.noarch.rpm
python-nova-2014.1.3-fuel5.1.1.mira7.git.ed5d120.7ad20ad.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.1-updates-stable-5174/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package nova has been built for project openstack/nova
Package version == 2014.1.3, package release == fuel5.1.1~mira5+git.ed5d120.7ad20ad

Changeset: https://review.fuel-infra.org/5174
project: openstack/nova
branch: openstack-ci/fuel-5.1.1-updates/2014.1.1
author: Roman Podoliaka
committer: Roman Podoliaka
subject: Compute: Catch binding failed exception while init host
status: patchset-created

Files placed on repository:
nova-ajax-console-proxy_2014.1.3-fuel5.1.1~mira5+git.ed5d120.7ad20ad_all.deb
nova-api-ec2_2014.1.3-fuel5.1.1~mira5+git.ed5d120.7ad20ad_all.deb
nova-api-metadata_2014.1.3-fuel5.1.1~mira5+git.ed5d120.7ad20ad_all.deb
nova-api-os-compute_2014.1.3-fuel5.1.1~mira5+git.ed5d120.7ad20ad_all.deb
nova-api-os-volume_2014.1.3-fuel5.1.1~mira5+git.ed5d120.7ad20ad_all.deb
nova-api_2014.1.3-fuel5.1.1~mira5+git.ed5d120.7ad20ad_all.deb
nova-baremetal_2014.1.3-fuel5.1.1~mira5+git.ed5d120.7ad20ad_all.deb
nova-cells_2014.1.3-fuel5.1.1~mira5+git.ed5d120.7ad20ad_all.deb
nova-cert_2014.1.3-fuel5.1.1~mira5+git.ed5d120.7ad20ad_all.deb
nova-common_2014.1.3-fuel5.1.1~mira5+git.ed5d120.7ad20ad_all.deb
nova-compute-kvm_2014.1.3-fuel5.1.1~mira5+git.ed5d120.7ad20ad_all.deb
nova-compute-libvirt_2014.1.3-fuel5.1.1~mira5+git.ed5d120.7ad20ad_all.deb
nova-compute-lxc_2014.1.3-fuel5.1.1~mira5+git.ed5d120.7ad20ad_all.deb
nova-compute-qemu_2014.1.3-fuel5.1.1~mira5+git.ed5d120.7ad20ad_all.deb
nova-compute-vmware_2014.1.3-fuel5.1.1~mira5+git.ed5d120.7ad20ad_all.deb
nova-compute-xen_2014.1.3-fuel5.1.1~mira5+git.ed5d120.7ad20ad_all.deb
nova-compute_2014.1.3-fuel5.1.1~mira5+git.ed5d120.7ad20ad_all.deb
nova-conductor_2014.1.3-fuel5.1.1~mira5+git.ed5d120.7ad20ad_all.deb
nova-console_2014.1.3-fuel5.1.1~mira5+git.ed5d120.7ad20ad_all.deb
nova-consoleauth_2014.1.3-fuel5.1.1~mira5+git.ed5d120.7ad20ad_all.deb
nova-doc_2014.1.3-fuel5.1.1~mira5+git.ed5d120.7ad20ad_all.deb
nova-network_2014.1.3-fuel5.1.1~mira5+git.ed5d120.7ad20ad_all.deb
nova-novncproxy_2014.1.3-fuel5.1.1~mira5+git.ed5d120.7ad20ad_all.deb
nova-objectstore_2014.1.3-fuel5.1.1~mira5+git.ed5d120.7ad20ad_all.deb
nova-scheduler_2014.1.3-fuel5.1.1~mira5+git.ed5d120.7ad20ad_all.deb
nova-spiceproxy_2014.1.3-fuel5.1.1~mira5+git.ed5d120.7ad20ad_all.deb
nova-volume_2014.1.3-fuel5.1.1~mira5+git.ed5d120.7ad20ad_all.deb
nova-xvpvncproxy_2014.1.3-fuel5.1.1~mira5+git.ed5d120.7ad20ad_all.deb
python-nova_2014.1.3-fuel5.1.1~mira5+git.ed5d120.7ad20ad_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.1-updates-stable-5174/ubuntu

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/nova (openstack-ci/fuel-6.0-updates/2014.2)

Fix proposed to branch: openstack-ci/fuel-6.0-updates/2014.2
Change author: Wangpan <email address hidden>
Review: https://review.fuel-infra.org/6436

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/nova (openstack-ci/fuel-6.0-updates/2014.2)

Reviewed: https://review.fuel-infra.org/6436
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-6.0-updates/2014.2

Commit: e087fd65fdc2d7b9999095c51982f7ec11aba3b1
Author: Wangpan <email address hidden>
Date: Wed May 6 10:30:21 2015

Compute: Catch binding failed exception while init host

While compute starts it will init all instances,
if an exception is raised from one instance
(e.g NovaException during plug_vifs), then the
compute process exits unexpectedly because of
this unhandled exception.
This commit changes the NovaException to more
appropriate VirtualInterfacePlugException and
catches it during init host, as well as the
instance is set to error state, with this change
the compute process can be started normally even
if this VirtualInterfacePlugException is raised.

Closes-bug: #1438680

Conflicts:
 nova/tests/unit/compute/test_compute_mgr.py

Change-Id: Ia584dba66affb86787e3069df19bd17b89cb5c49
(cherry picked from commit 16ac50b1e760b7d20b840763b271a497b66ad5a5)
(cherry picked from commit d930a43f388359e689b05beb675ed83e50b784c6)

Pavel Kholkin (pkholkin)
tags: added: on-verification
Revision history for this message
Pavel Kholkin (pkholkin) wrote :

Verified on MOS 6.1 ISO #435

Steps to Verify:

1. Boot active vm
2. Insert 'raise exception.VirtualInterfacePlugException()' to 'def plug(self, instance, vif)' in '/usr/lib/python2.7/dist-packages/nova/virt/libvirt/vif.py' file.
3. restart nova-compute service on compute node
4. check that nova-compute service is still alive

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/nova (openstack-ci/fuel-5.1-updates/2014.1.1)

Fix proposed to branch: openstack-ci/fuel-5.1-updates/2014.1.1
Change author: Alex Ermolov <email address hidden>
Review: https://review.fuel-infra.org/8917

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on openstack/nova (openstack-ci/fuel-5.1.1-updates/2014.1.1)

Change abandoned by Alex Ermolov <email address hidden> on branch: openstack-ci/fuel-5.1.1-updates/2014.1.1
Review: https://review.fuel-infra.org/5174

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change restored on openstack/nova (openstack-ci/fuel-5.1.1-updates/2014.1.1)

Change restored by Alex Ermolov <email address hidden> on branch: openstack-ci/fuel-5.1.1-updates/2014.1.1
Review: https://review.fuel-infra.org/5174

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on openstack/nova (openstack-ci/fuel-5.1.1-updates/2014.1.1)

Change abandoned by Alex Ermolov <email address hidden> on branch: openstack-ci/fuel-5.1.1-updates/2014.1.1
Review: https://review.fuel-infra.org/5174

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/nova (openstack-ci/fuel-5.1.1-updates/2014.1.1)

Fix proposed to branch: openstack-ci/fuel-5.1.1-updates/2014.1.1
Change author: Alex Ermolov <email address hidden>
Review: https://review.fuel-infra.org/9384

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/nova (openstack-ci/fuel-5.1-updates/2014.1.1)

Reviewed: https://review.fuel-infra.org/8917
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-5.1-updates/2014.1.1

Commit: 26d94804952e4219892bc64dd654005a10b5496e
Author: Alex Ermolov <email address hidden>
Date: Wed Jul 15 16:01:53 2015

Compute: Catch binding failed exception while init host

While compute starts it will init all instances,
if an exception is raised from one instance
(e.g NovaException during plug_vifs), then the
compute process exits unexpectedly because of
this unhandled exception.
This commit changes the NovaException to more
appropriate VirtualInterfacePlugException and
catches it during init host, as well as the
instance is set to error state, with this change
the compute process can be started normally even
if this VirtualInterfacePlugException is raised.

Closes-bug: #1438680
Change-Id: Ia584dba66affb86787e3069df19bd17b89cb5c49

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/nova (openstack-ci/fuel-5.1.1-updates/2014.1.1)

Reviewed: https://review.fuel-infra.org/9384
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-5.1.1-updates/2014.1.1

Commit: a1e4b5343b43dfef4d8ba55c1f48fae21752072c
Author: Alex Ermolov <email address hidden>
Date: Mon Jul 20 08:37:32 2015

Compute: Catch binding failed exception while init host

While compute starts it will init all instances,
if an exception is raised from one instance
(e.g NovaException during plug_vifs), then the
compute process exits unexpectedly because of
this unhandled exception.
This commit changes the NovaException to more
appropriate VirtualInterfacePlugException and
catches it during init host, as well as the
instance is set to error state, with this change
the compute process can be started normally even
if this VirtualInterfacePlugException is raised.

Closes-bug: #1438680

Conflicts:
 nova/tests/unit/compute/test_compute_mgr.py

Change-Id: I4784773d3e23f42c7d06719b53337546c3e4ea48

Revision history for this message
Vadim Rovachev (vrovachev) wrote :

verified on 5.1 and 5.1.1 MOS versions

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.