Build notification in conductor fails to send due to InstanceNotFound

Bug #1721670 reported by melanie witt on 2017-10-05
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
High
melanie witt
Ocata
High
melanie witt
Pike
High
melanie witt

Bug Description

I found this issue while working on improving the CellDatabases fixture by defaulting untargeted DB access to 'cell0' instead of 'cell1'.

While building an instance in conductor, it sends a notification about the changed state using notifications.send_update_with_states. In notifications.send_update_with_states, one of the arguments is an Instance and if an attribute needs to be lazy-loaded, (example: tags, as part of the InstanceUpdatePayload) and the load method contains a _check_instance_exists_in_project call in the DB layer, InstanceNotFound is raised at that point because the context wasn't targeted to the instance's cell.

We need to target the context in case the notification call needs to load something from the instance's cell database.

Fix proposed to branch: master
Review: https://review.openstack.org/509967

Changed in nova:
status: New → In Progress

Reviewed: https://review.openstack.org/509967
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=54cf9f5a90d745896d133669e6d1260e6adee51b
Submitter: Jenkins
Branch: master

commit 54cf9f5a90d745896d133669e6d1260e6adee51b
Author: melanie witt <email address hidden>
Date: Thu Oct 5 23:55:58 2017 +0000

    Target context for build notification in conductor

    When sending notifications about an instance, it's possible the
    notification payload will have attributes not already loaded on
    the instance. In this case, a lazy-load will happen and if the
    load method needs to lookup the instance (example: in the
    _check_instance_exists_in_project method in the DB layer), the
    InstanceNotFound exception will be raised and the notification
    will fail to be sent.

    This targets the instance context to the instance's cell when
    calling the send notification method.

    Closes-Bug: #1721670

    Change-Id: I6b9eb120b6e7fcbf919a3791afe2d1f6a7bd3b60

Changed in nova:
status: In Progress → Fix Released
Matt Riedemann (mriedem) on 2017-10-11
Changed in nova:
importance: Undecided → High
melanie witt (melwitt) on 2017-10-11
Changed in nova:
importance: High → Medium
Matt Riedemann (mriedem) on 2017-10-11
tags: added: notifications
melanie witt (melwitt) on 2017-10-11
Changed in nova:
importance: Medium → High

Reviewed: https://review.openstack.org/511292
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=594b33469bc162174eb2e221849b2aac78a5d130
Submitter: Zuul
Branch: stable/pike

commit 594b33469bc162174eb2e221849b2aac78a5d130
Author: melanie witt <email address hidden>
Date: Thu Oct 5 23:55:58 2017 +0000

    Target context for build notification in conductor

    When sending notifications about an instance, it's possible the
    notification payload will have attributes not already loaded on
    the instance. In this case, a lazy-load will happen and if the
    load method needs to lookup the instance (example: in the
    _check_instance_exists_in_project method in the DB layer), the
    InstanceNotFound exception will be raised and the notification
    will fail to be sent.

    This targets the instance context to the instance's cell when
    calling the send notification method.

    Closes-Bug: #1721670

    Change-Id: I6b9eb120b6e7fcbf919a3791afe2d1f6a7bd3b60
    (cherry picked from commit 54cf9f5a90d745896d133669e6d1260e6adee51b)

Reviewed: https://review.openstack.org/511325
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d7a60f8a1c51070237590d975b1acc61f4bbcfca
Submitter: Zuul
Branch: stable/ocata

commit d7a60f8a1c51070237590d975b1acc61f4bbcfca
Author: melanie witt <email address hidden>
Date: Thu Oct 5 23:55:58 2017 +0000

    Target context for build notification in conductor

    When sending notifications about an instance, it's possible the
    notification payload will have attributes not already loaded on
    the instance. In this case, a lazy-load will happen and if the
    load method needs to lookup the instance (example: in the
    _check_instance_exists_in_project method in the DB layer), the
    InstanceNotFound exception will be raised and the notification
    will fail to be sent.

    This targets the instance context to the instance's cell when
    calling the send notification method.

    Closes-Bug: #1721670

     Conflicts:
     nova/conductor/manager.py
     nova/tests/unit/conductor/test_conductor.py

    NOTE(melwitt): Conflict was due to the older version of target_cell
    not yielding a new RequestContext, the absence of set_target_cell,
    and an older version of the unit test in Ocata.

    Change-Id: I6b9eb120b6e7fcbf919a3791afe2d1f6a7bd3b60
    (cherry picked from commit 54cf9f5a90d745896d133669e6d1260e6adee51b)

This issue was fixed in the openstack/nova 17.0.0.0b1 development milestone.

This issue was fixed in the openstack/nova 16.0.2 release.

This issue was fixed in the openstack/nova 15.0.8 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers