Don't trace when @reverts_task_state fails on InstanceNotFound

Bug #1431404 reported by Matt Riedemann
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Matt Riedemann
Juno
Fix Released
Undecided
Unassigned

Bug Description

This change https://review.openstack.org/#/c/163515/ added a warning when the @reverts_task_state decorator in the compute manager fails rather than just pass, because we were getting KeyErrors and never noticing them which broke the decorator.

However, now we're tracing on InstanceNotFound which is a normal case if we're deleting the instance after a failure (tempest will delete the instance immediately after failures when tearing down a test):

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiRmFpbGVkIHRvIHJldmVydCB0YXNrIHN0YXRlIGZvciBpbnN0YW5jZVwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI4NjQwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MjYxNzA3MDE2OTV9

http://logs.openstack.org/98/163798/1/check/check-tempest-dsvm-postgres-full/6eff665/logs/screen-n-cpu.txt.gz#_2015-03-12_13_11_36_304

2015-03-12 13:11:36.304 WARNING nova.compute.manager [req-a5f3b37e-19e9-4e1d-9be7-bbb9a8e7f4c1 DeleteServersTestJSON-706956764 DeleteServersTestJSON-535578435] [instance: 6de2ad51-3155-4538-830d-f02de39b4be3] Failed to revert task state for instance. Error: Instance 6de2ad51-3155-4538-830d-f02de39b4be3 could not be found.
Traceback (most recent call last):

  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 142, in inner
    return func(*args, **kwargs)

  File "/opt/stack/new/nova/nova/conductor/manager.py", line 134, in instance_update
    columns_to_join=['system_metadata'])

  File "/opt/stack/new/nova/nova/db/api.py", line 774, in instance_update_and_get_original
    columns_to_join=columns_to_join)

  File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 143, in wrapper
    return f(*args, **kwargs)

  File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 2395, in instance_update_and_get_original
    columns_to_join=columns_to_join)

  File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 181, in wrapped
    return f(*args, **kwargs)

  File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 2434, in _instance_update
    columns_to_join=columns_to_join)

  File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 1670, in _instance_get_by_uuid
    raise exception.InstanceNotFound(instance_id=uuid)

InstanceNotFound: Instance 6de2ad51-3155-4538-830d-f02de39b4be3 could not be found.

Matt Riedemann (mriedem)
tags: added: icehouse-backport-potential juno-backport-potential
removed: junot
Changed in nova:
assignee: nobody → Matt Riedemann (mriedem)
importance: Undecided → Medium
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/163890

Matt Riedemann (mriedem)
Changed in nova:
milestone: none → kilo-3
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/163890
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=95976ca1aff63c01ed119c9a3c39927f39ce1fbf
Submitter: Jenkins
Branch: master

commit 95976ca1aff63c01ed119c9a3c39927f39ce1fbf
Author: Matt Riedemann <email address hidden>
Date: Thu Mar 12 09:09:49 2015 -0700

    compute: don't trace on InstanceNotFound in reverts_task_state

    Commit c43f2b0d708f0f4b37850d2917c0abcc13b8789b added logging when
    _instance_update fails in reverts_task_state but we shouldn't log
    InstanceNotFound since it's a normal (expected) error when we're
    deleting an instance shortly after it fails to build.

    Closes-Bug: #1431404

    Change-Id: Iec3dfaa16b472bc88d56d9c6680a7c247f2f50bd

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/165189

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/icehouse)

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/165219

Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/icehouse)

Change abandoned by Matt Riedemann (<email address hidden>) on branch: stable/icehouse
Review: https://review.openstack.org/165219
Reason: Not a security fix so abandoning.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/juno)

Reviewed: https://review.openstack.org/165189
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5ff5afe7df8e296f03fd9fd86ad169360cc91451
Submitter: Jenkins
Branch: stable/juno

commit 5ff5afe7df8e296f03fd9fd86ad169360cc91451
Author: Matt Riedemann <email address hidden>
Date: Thu Mar 12 09:09:49 2015 -0700

    compute: don't trace on InstanceNotFound in reverts_task_state

    Commit c43f2b0d708f0f4b37850d2917c0abcc13b8789b added logging when
    _instance_update fails in reverts_task_state but we shouldn't log
    InstanceNotFound since it's a normal (expected) error when we're
    deleting an instance shortly after it fails to build.

    Closes-Bug: #1431404

    Conflicts:
            nova/tests/unit/compute/test_compute_mgr.py

    NOTE(mriedem): The conflict is due to the test modules being
    moved in Kilo and the need to import nova.compute.manager.

    Change-Id: Iec3dfaa16b472bc88d56d9c6680a7c247f2f50bd
    (cherry picked from commit 95976ca1aff63c01ed119c9a3c39927f39ce1fbf)

tags: added: in-stable-juno
Thierry Carrez (ttx)
Changed in nova:
milestone: kilo-3 → 2015.1.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.