Build notification in conductor fails to send due to InstanceNotFound

Bug #1721670 reported by melanie witt
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
melanie witt
Ocata
Fix Committed
High
melanie witt
Pike
Fix Committed
High
melanie witt

Bug Description

I found this issue while working on improving the CellDatabases fixture by defaulting untargeted DB access to 'cell0' instead of 'cell1'.

While building an instance in conductor, it sends a notification about the changed state using notifications.send_update_with_states. In notifications.send_update_with_states, one of the arguments is an Instance and if an attribute needs to be lazy-loaded, (example: tags, as part of the InstanceUpdatePayload) and the load method contains a _check_instance_exists_in_project call in the DB layer, InstanceNotFound is raised at that point because the context wasn't targeted to the instance's cell.

We need to target the context in case the notification call needs to load something from the instance's cell database.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/509967

Changed in nova:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/509967
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=54cf9f5a90d745896d133669e6d1260e6adee51b
Submitter: Jenkins
Branch: master

commit 54cf9f5a90d745896d133669e6d1260e6adee51b
Author: melanie witt <email address hidden>
Date: Thu Oct 5 23:55:58 2017 +0000

    Target context for build notification in conductor

    When sending notifications about an instance, it's possible the
    notification payload will have attributes not already loaded on
    the instance. In this case, a lazy-load will happen and if the
    load method needs to lookup the instance (example: in the
    _check_instance_exists_in_project method in the DB layer), the
    InstanceNotFound exception will be raised and the notification
    will fail to be sent.

    This targets the instance context to the instance's cell when
    calling the send notification method.

    Closes-Bug: #1721670

    Change-Id: I6b9eb120b6e7fcbf919a3791afe2d1f6a7bd3b60

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/511292

Matt Riedemann (mriedem)
Changed in nova:
importance: Undecided → High
melanie witt (melwitt)
Changed in nova:
importance: High → Medium
Matt Riedemann (mriedem)
tags: added: notifications
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/511325

melanie witt (melwitt)
Changed in nova:
importance: Medium → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/pike)

Reviewed: https://review.openstack.org/511292
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=594b33469bc162174eb2e221849b2aac78a5d130
Submitter: Zuul
Branch: stable/pike

commit 594b33469bc162174eb2e221849b2aac78a5d130
Author: melanie witt <email address hidden>
Date: Thu Oct 5 23:55:58 2017 +0000

    Target context for build notification in conductor

    When sending notifications about an instance, it's possible the
    notification payload will have attributes not already loaded on
    the instance. In this case, a lazy-load will happen and if the
    load method needs to lookup the instance (example: in the
    _check_instance_exists_in_project method in the DB layer), the
    InstanceNotFound exception will be raised and the notification
    will fail to be sent.

    This targets the instance context to the instance's cell when
    calling the send notification method.

    Closes-Bug: #1721670

    Change-Id: I6b9eb120b6e7fcbf919a3791afe2d1f6a7bd3b60
    (cherry picked from commit 54cf9f5a90d745896d133669e6d1260e6adee51b)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/ocata)

Reviewed: https://review.openstack.org/511325
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d7a60f8a1c51070237590d975b1acc61f4bbcfca
Submitter: Zuul
Branch: stable/ocata

commit d7a60f8a1c51070237590d975b1acc61f4bbcfca
Author: melanie witt <email address hidden>
Date: Thu Oct 5 23:55:58 2017 +0000

    Target context for build notification in conductor

    When sending notifications about an instance, it's possible the
    notification payload will have attributes not already loaded on
    the instance. In this case, a lazy-load will happen and if the
    load method needs to lookup the instance (example: in the
    _check_instance_exists_in_project method in the DB layer), the
    InstanceNotFound exception will be raised and the notification
    will fail to be sent.

    This targets the instance context to the instance's cell when
    calling the send notification method.

    Closes-Bug: #1721670

     Conflicts:
     nova/conductor/manager.py
     nova/tests/unit/conductor/test_conductor.py

    NOTE(melwitt): Conflict was due to the older version of target_cell
    not yielding a new RequestContext, the absence of set_target_cell,
    and an older version of the unit test in Ocata.

    Change-Id: I6b9eb120b6e7fcbf919a3791afe2d1f6a7bd3b60
    (cherry picked from commit 54cf9f5a90d745896d133669e6d1260e6adee51b)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 17.0.0.0b1

This issue was fixed in the openstack/nova 17.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 16.0.2

This issue was fixed in the openstack/nova 16.0.2 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 15.0.8

This issue was fixed in the openstack/nova 15.0.8 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.