Improve error log when instance snapshot fails

Bug #1613770 reported by vu tran
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Low
vu tran
Queens
New
Undecided
Unassigned
Rocky
New
Undecided
Unassigned
Stein
Fix Released
Undecided
Unassigned
Train
Fix Released
Undecided
Unassigned

Bug Description

If glance backend store is set to use filesystem storage and this storage is running out of space when if glance is trying to create a instance snapshot, then in nova-compute log the following message is displayed:

2016-08-08 22:24:31.644 TRACE oslo_messaging.rpc.server HTTPOverLimit: 413 Request Entity Too Large
2016-08-08 22:24:31.644 TRACE oslo_messaging.rpc.server Image storage media is full: There is not enough disk space on the image storage media.
2016-08-08 22:24:31.644 TRACE oslo_messaging.rpc.server (HTTP 413)

It's a little bit annoying that we're logging the HTTP error from glance and that we don't specify the image uuid.

Steps to reproduce:
* set glance's config filesystem_store_datadir to small size filesystem
* start nova instance
* keep invoking "nova image-create" to create instance image snapshot, eventually the backend filesystem storage would run out of space
* on nova-compute log see the HTTP error message above.

vu tran (vu-tran)
Changed in nova:
assignee: nobody → vu tran (vu-tran)
Changed in nova:
status: New → In Progress
Revision history for this message
Sean Dague (sdague) wrote :

There are no currently open reviews on this bug, changing
the status back to the previous state and unassigning. If
there are active reviews related to this bug, please include
links in comments.

Changed in nova:
status: In Progress → New
assignee: vu tran (vu-tran) → nobody
Revision history for this message
Sean Dague (sdague) wrote :

Found open reviews for this bug in gerrit, setting to In Progress.

review: https://review.openstack.org/356020 in branch: master

Changed in nova:
status: New → In Progress
assignee: nobody → vu tran (vu-tran)
Changed in nova:
assignee: vu tran (vu-tran) → Stephen Finucane (stephenfinucane)
Changed in nova:
assignee: Stephen Finucane (stephenfinucane) → Matt Riedemann (mriedem)
Matt Riedemann (mriedem)
Changed in nova:
importance: Undecided → Low
assignee: Matt Riedemann (mriedem) → vu tran (vu-tran)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.opendev.org/356020
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=024bf10d8aec5e58111793a9652b16682eb61b7c
Submitter: Zuul
Branch: master

commit 024bf10d8aec5e58111793a9652b16682eb61b7c
Author: Vu Tran <email address hidden>
Date: Mon Aug 29 16:33:19 2016 -0400

    Improve error log when snapshot fails

    If snapshot creation via glance fails due to lack of space or over
    quota, we want to have a clearer error message.

    Change-Id: Ic9133f6bc14d4fe766d37a438bf52c33e89da768
    Closes-Bug: #1613770

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/train)

Reviewed: https://review.opendev.org/c/openstack/nova/+/782982
Committed: https://opendev.org/openstack/nova/commit/9e9c022bde3a3ffdf0dd87e21bf9afde0dbc1e74
Submitter: "Zuul (22348)"
Branch: stable/train

commit 9e9c022bde3a3ffdf0dd87e21bf9afde0dbc1e74
Author: Vu Tran <email address hidden>
Date: Mon Aug 29 16:33:19 2016 -0400

    Improve error log when snapshot fails

    If snapshot creation via glance fails due to lack of space or over
    quota, we want to have a clearer error message.

    Change-Id: Ic9133f6bc14d4fe766d37a438bf52c33e89da768
    Closes-Bug: #1613770
    (cherry picked from commit 024bf10d8aec5e58111793a9652b16682eb61b7c)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 20.6.1

This issue was fixed in the openstack/nova 20.6.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/stein)

Reviewed: https://review.opendev.org/c/openstack/nova/+/782962
Committed: https://opendev.org/openstack/nova/commit/446c9c56109329208193e3c9b106a0a2eb3cd9ce
Submitter: "Zuul (22348)"
Branch: stable/stein

commit 446c9c56109329208193e3c9b106a0a2eb3cd9ce
Author: Vu Tran <email address hidden>
Date: Mon Aug 29 16:33:19 2016 -0400

    Improve error log when snapshot fails

    If snapshot creation via glance fails due to lack of space or over
    quota, we want to have a clearer error message.

    Change-Id: Ic9133f6bc14d4fe766d37a438bf52c33e89da768
    Closes-Bug: #1613770
    (cherry picked from commit 024bf10d8aec5e58111793a9652b16682eb61b7c)
    (cherry picked from commit 9e9c022bde3a3ffdf0dd87e21bf9afde0dbc1e74)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/queens)

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/queens
Review: https://review.opendev.org/c/openstack/nova/+/782984
Reason: This branch transitioned to End of Life for this project, open patches needs to be closed to be able to delete the branch.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova stein-eol

This issue was fixed in the openstack/nova stein-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/rocky)

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/rocky
Review: https://review.opendev.org/c/openstack/nova/+/782983
Reason: This branch transitioned to End of Life for this project, open patches needs to be closed to be able to delete the branch.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.