resume guests on libvirt host reboot fails for instances created from multi-part image

Bug #1462991 reported by Alex Ermolov
20
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Fix Released
High
MOS Nova
5.1.x
Fix Committed
High
Denis Meltsaykin
6.0.x
Fix Released
High
Alex Ermolov
6.1.x
Fix Released
High
Alex Ermolov
7.0.x
Fix Released
High
MOS Nova

Bug Description

This is backport for upstream fix for: https://bugs.launchpad.net/nova/+bug/1195884

Original description:
=====================

If an instance is created from a multi-part image (i.e. separate kernel and ramdisk files) , and resume_guests_state_on_host_boot=True the resulting _hard_reboot will fail and put the instance into an error state.

This is because the code in _hard_reboot that checks the image files, _create_images_and_backing() always tries to reload these from Glance.

In a normal reboot this results in an unnecessary image download from Glance.

In the case of reboot following a host boot the download fails because the context at this point does not have credentials to be able to access Glance.

Possible fix is to:
- Check files exist rather than always trying to download them
- Provide in conf credentials of a user that has read access to all images in Glance than can be used in this context

Alex Ermolov (aermolov)
Changed in mos:
importance: Low → High
assignee: nobody → Alex Ermolov (aermolov)
Changed in mos:
status: New → Confirmed
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/nova (openstack-ci/fuel-6.1/2014.2)

Fix proposed to branch: openstack-ci/fuel-6.1/2014.2
Change author: Alex Ermolov <email address hidden>
Review: https://review.fuel-infra.org/7667

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/nova (openstack-ci/fuel-6.0-updates/2014.2)

Fix proposed to branch: openstack-ci/fuel-6.0-updates/2014.2
Change author: Alex Ermolov <email address hidden>
Review: https://review.fuel-infra.org/7716

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/nova (openstack-ci/fuel-6.0-updates/2014.2)

Reviewed: https://review.fuel-infra.org/7716
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-6.0-updates/2014.2

Commit: 988c7be9958abd1b88f282ed768f513038804450
Author: Alex Ermolov <email address hidden>
Date: Wed Jun 10 15:09:32 2015

libvirt: Handle empty context on _hard_reboot

When _hard_reboot runs for resume_guest_state_on_host_boot, we don't
have a valid context. This checks if the context is valid before
attempting to repopulate base images.

This is a bit of a hack, as it assumes that the base images already
exist on the system, and will fail if they don't. However, it reduces
the number of failures (without this, every downed guest is guaranteed
to error when coming back up for resume_guest_state_on_host_boot).

Closes-Bug: 1462991
Change-Id: Ibac52f5d1b4e5ab2adb0a561659e6df8c7c1f40f

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to patching-tests (master)

Fix proposed to branch: master
Change author: Denis V. Meltsaykin <email address hidden>
Review: https://review.fuel-infra.org/7985

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on patching-tests (master)

Change abandoned by Denis V. Meltsaykin <email address hidden> on branch: master
Review: https://review.fuel-infra.org/7985

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to patching-tests (stable/6.1)

Fix proposed to branch: stable/6.1
Change author: Denis V. Meltsaykin <email address hidden>
Review: https://review.fuel-infra.org/8628

Alexey Khivin (akhivin)
tags: added: 6.1-mu-1
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to patching-tests (stable/6.1)

Reviewed: https://review.fuel-infra.org/8628
Submitter: Alex Khivin <email address hidden>
Branch: stable/6.1

Commit: f24dcd76db197c0d62f8a585f70e813446eba25b
Author: Denis V. Meltsaykin <email address hidden>
Date: Wed Jul 1 14:11:16 2015

libvirt: Handle empty context on _hard_reboot

When _hard_reboot runs for resume_guest_state_on_host_boot,
we don't have a valid context. This checks if the context
is valid before attempting to repopulate base images.
This is a bit of a hack, as it assumes that the base images
already exist on the system, and will fail if they don't.
However, it reduces the number of failures (without this,
every downed guest is guaranteed to error when coming back
up for resume_guest_state_on_host_boot).

Change-Id: I82db6963f9cbfa2ce103e11754cb53a5a7bd2aa1
Closes-Bug: 1462991

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to patching-tests (stable/6.1)

Fix proposed to branch: stable/6.1
Change author: Denis V. Meltsaykin <email address hidden>
Review: https://review.fuel-infra.org/8967

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to patching-tests (stable/6.1)

Reviewed: https://review.fuel-infra.org/8967
Submitter: Alex Khivin <email address hidden>
Branch: stable/6.1

Commit: 42bb875b43e65996598898015c7515e3ff96388a
Author: Denis V. Meltsaykin <email address hidden>
Date: Thu Jul 2 13:44:59 2015

libvirt: Handle empty context on _hard_reboot

When _hard_reboot runs for resume_guest_state_on_host_boot,
we don't have a valid context. This checks if the context is
valid before attempting to repopulate base images.
This is a bit of a hack, as it assumes that the base images
already exist on the system, and will fail if they don't.
However, it reduces the number of failures (without this,
every downed guest is guaranteed to error when coming back up for
resume_guest_state_on_host_boot).

Change-Id: Idcc973b049b78e1ac4f4dd3177fee899340b86f9
Partial-Bug: 1462991

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix proposed to patching-tests (stable/6.1)

Related fix proposed to branch: stable/6.1
Change author: Denis V. Meltsaykin <email address hidden>
Review: https://review.fuel-infra.org/8970

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix merged to patching-tests (stable/6.1)

Reviewed: https://review.fuel-infra.org/8970
Submitter: Alex Khivin <email address hidden>
Branch: stable/6.1

Commit: 26520c6439350251826468997f31f10b151e0c46
Author: Denis V. Meltsaykin <email address hidden>
Date: Thu Jul 2 14:17:08 2015

Fix for 1462991

Fixing erratum to tests run better

Change-Id: Ibb3ce73283e6cb6d05d22242a1adac538fbda89c
Related-Bug: 1462991

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to patching-tests (stable/6.1)

Reviewed: https://review.fuel-infra.org/9116
Submitter: Alexey Shtokolov <email address hidden>
Branch: stable/6.1

Commit: 480947d7dedd2d9f6362eb2c1a7ec540d0ffde27
Author: Vadim Rovachev <email address hidden>
Date: Tue Jul 7 15:27:24 2015

Add workaround for centos distro for glance part images bug

Added one more time reboot for computes for reproduce bug
with error instance state and part images.

Closes-Bug: #1462991

Change-Id: If159fbf9889a34ce771c66f3a520ceeb627c989f

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/nova (openstack-ci/fuel-6.1/2014.2)

Reviewed: https://review.fuel-infra.org/7667
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-6.1/2014.2

Commit: 426d17259ad75c4e9180cb7452afa45062b47d01
Author: Alex Ermolov <email address hidden>
Date: Tue Jun 9 16:28:40 2015

libvirt: Handle empty context on _hard_reboot

When _hard_reboot runs for resume_guest_state_on_host_boot, we don't
have a valid context. This checks if the context is valid before
attempting to repopulate base images.

This is a bit of a hack, as it assumes that the base images already
exist on the system, and will fail if they don't. However, it reduces
the number of failures (without this, every downed guest is guaranteed
to error when coming back up for resume_guest_state_on_host_boot).

Closes-Bug: 1462991
Change-Id: Ibac52f5d1b4e5ab2adb0a561659e6df8c7c1f40f

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix merged to patching-tests (stable/6.1)

Reviewed: https://review.fuel-infra.org/9161
Submitter: Vitaly Sedelnik <email address hidden>
Branch: stable/6.1

Commit: 5c4de400d8281940905a2a6bcb583ae5b987322c
Author: Denis V. Meltsaykin <email address hidden>
Date: Wed Jul 8 17:09:18 2015

Change in 1462991's description

Related-Bug: #1462991

Change-Id: Ib7e93e43b75ec589db162f8d74665b96b6daa79a

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix proposed to patching-tests (stable/6.1)

Related fix proposed to branch: stable/6.1
Change author: Denis V. Meltsaykin <email address hidden>
Review: https://review.fuel-infra.org/9217

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix merged to patching-tests (stable/6.1)

Reviewed: https://review.fuel-infra.org/9217
Submitter: Vitaly Sedelnik <email address hidden>
Branch: stable/6.1

Commit: 478bfc270fda2bc7e870c311bec645885659b828
Author: Denis V. Meltsaykin <email address hidden>
Date: Fri Jul 10 10:45:11 2015

Small description/title changes

Change-Id: I13cbc5a659ece19791ce6c7ed1c1823ebcaf2fb5
Related-Bug: #1462991

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on openstack/nova (openstack-ci/fuel-5.1.1-updates/2014.1.1)

Change abandoned by Alex Ermolov <email address hidden> on branch: openstack-ci/fuel-5.1.1-updates/2014.1.1
Review: https://review.fuel-infra.org/8901

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change restored on openstack/nova (openstack-ci/fuel-5.1.1-updates/2014.1.1)

Change restored by Alex Ermolov <email address hidden> on branch: openstack-ci/fuel-5.1.1-updates/2014.1.1
Review: https://review.fuel-infra.org/8901

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/nova (openstack-ci/fuel-5.1.1-updates/2014.1.1)

Fix proposed to branch: openstack-ci/fuel-5.1.1-updates/2014.1.1
Change author: Alex Ermolov <email address hidden>
Review: https://review.fuel-infra.org/9383

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on openstack/nova (openstack-ci/fuel-5.1.1-updates/2014.1.1)

Change abandoned by Alex Ermolov <email address hidden> on branch: openstack-ci/fuel-5.1.1-updates/2014.1.1
Review: https://review.fuel-infra.org/8901

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/nova (openstack-ci/fuel-5.1-updates/2014.1.1)

Fix proposed to branch: openstack-ci/fuel-5.1-updates/2014.1.1
Change author: Alex Ermolov <email address hidden>
Review: https://review.fuel-infra.org/9393

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/nova (openstack-ci/fuel-5.1-updates/2014.1.1)

Reviewed: https://review.fuel-infra.org/9393
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-5.1-updates/2014.1.1

Commit: 1dec25b1fec7ddb29d08e390999dbfbbcdf70863
Author: Alex Ermolov <email address hidden>
Date: Wed Jul 15 16:09:52 2015

libvirt: Handle empty context on _hard_reboot

When _hard_reboot runs for resume_guest_state_on_host_boot, we don't
have a valid context. This checks if the context is valid before
attempting to repopulate base images.

This is a bit of a hack, as it assumes that the base images already
exist on the system, and will fail if they don't. However, it reduces
the number of failures (without this, every downed guest is guaranteed
to error when coming back up for resume_guest_state_on_host_boot).

Closes-Bug: #1462991
Change-Id: Id20e73c19543e3f4e240eee5c78e3f13ba466a2f

Changed in mos:
status: In Progress → Confirmed
assignee: Alex Ermolov (aermolov) → MOS Nova (mos-nova)
milestone: 6.1-updates → 7.0
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/nova (openstack-ci/fuel-5.1.1-updates/2014.1.1)

Reviewed: https://review.fuel-infra.org/9383
Submitter: Denis V. Meltsaykin <email address hidden>
Branch: openstack-ci/fuel-5.1.1-updates/2014.1.1

Commit: d08dde790374aafde2a78663fc2cee0df9c1f065
Author: Alex Ermolov <email address hidden>
Date: Mon Nov 16 21:28:52 2015

libvirt: Handle empty context on _hard_reboot

When _hard_reboot runs for resume_guest_state_on_host_boot, we don't
have a valid context. This checks if the context is valid before
attempting to repopulate base images.

This is a bit of a hack, as it assumes that the base images already
exist on the system, and will fail if they don't. However, it reduces
the number of failures (without this, every downed guest is guaranteed
to error when coming back up for resume_guest_state_on_host_boot).

Change-Id: Id20e73c19543e3f4e240eee5c78e3f13ba466a2f
Closes-Bug: #1462991

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.