MigrationNotFound in multi-cell setup doing server external events processing

[Matt Riedemann]

I noticed this in the devstack change testing multi-cell, in the neutron multinode job:

http://logs.openstack.org/56/477556/3/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/ee3e9b6/logs/screen-n-api.txt.gz?level=TRACE#_Jul_05_20_55_53_749696

We're getting migration not found because we're not targeted to a cell to find the migration:

Jul 05 20:55:53.749696 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 <email address hidden>[27811]: ERROR nova.api.openstack.extensions [None req-d4ca5b2e-140c-4331-88eb-50bae2de2230 service nova] Unexpected exception in API method: MigrationNotFound: Migration 15 could not be found.
Jul 05 20:55:53.749806 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 <email address hidden>[27811]: ERROR nova.api.openstack.extensions Traceback (most recent call last):
Jul 05 20:55:53.749888 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 <email address hidden>[27811]: ERROR nova.api.openstack.extensions File "/opt/stack/new/nova/nova/api/openstack/extensions.py", line 336, in wrapped
Jul 05 20:55:53.749965 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 <email address hidden>[27811]: ERROR nova.api.openstack.extensions return f(*args, **kwargs)
Jul 05 20:55:53.750053 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 <email address hidden>[27811]: ERROR nova.api.openstack.extensions File "/opt/stack/new/nova/nova/api/validation/__init__.py", line 108, in wrapper
Jul 05 20:55:53.750134 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 <email address hidden>[27811]: ERROR nova.api.openstack.extensions return func(*args, **kwargs)
Jul 05 20:55:53.750222 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 <email address hidden>[27811]: ERROR nova.api.openstack.extensions File "/opt/stack/new/nova/nova/api/openstack/compute/server_external_events.py", line 120, in create
Jul 05 20:55:53.750303 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 <email address hidden>[27811]: ERROR nova.api.openstack.extensions context, accepted_instances, mappings, accepted_events)
Jul 05 20:55:53.750438 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 <email address hidden>[27811]: ERROR nova.api.openstack.extensions File "/opt/stack/new/nova/nova/compute/api.py", line 4046, in external_instance_event
Jul 05 20:55:53.750564 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 <email address hidden>[27811]: ERROR nova.api.openstack.extensions for host in self._get_relevant_hosts(context, instance):
Jul 05 20:55:53.750647 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 <email address hidden>[27811]: ERROR nova.api.openstack.extensions File "/opt/stack/new/nova/nova/compute/api.py", line 4072, in _get_relevant_hosts
Jul 05 20:55:53.750722 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 <email address hidden>[27811]: ERROR nova.api.openstack.extensions migration = objects.Migration.get_by_id(context, migration_id)
Jul 05 20:55:53.750798 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 <email address hidden>[27811]: ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
Jul 05 20:55:53.750873 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 <email address hidden>[27811]: ERROR nova.api.openstack.extensions result = fn(cls, context, *args, **kwargs)
Jul 05 20:55:53.750949 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 <email address hidden>[27811]: ERROR nova.api.openstack.extensions File "/opt/stack/new/nova/nova/objects/migration.py", line 106, in get_by_id
Jul 05 20:55:53.751023 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 <email address hidden>[27811]: ERROR nova.api.openstack.extensions db_migration = db.migration_get(context, migration_id)
Jul 05 20:55:53.751107 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 <email address hidden>[27811]: ERROR nova.api.openstack.extensions File "/opt/stack/new/nova/nova/db/api.py", line 535, in migration_get
Jul 05 20:55:53.751187 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 <email address hidden>[27811]: ERROR nova.api.openstack.extensions return IMPL.migration_get(context, migration_id)
Jul 05 20:55:53.751264 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 <email address hidden>[27811]: ERROR nova.api.openstack.extensions File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 265, in wrapped
Jul 05 20:55:53.751338 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 <email address hidden>[27811]: ERROR nova.api.openstack.extensions return f(context, *args, **kwargs)
Jul 05 20:55:53.751414 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 <email address hidden>[27811]: ERROR nova.api.openstack.extensions File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 4719, in migration_get
Jul 05 20:55:53.751492 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 <email address hidden>[27811]: ERROR nova.api.openstack.extensions raise exception.MigrationNotFound(migration_id=id)
Jul 05 20:55:53.751568 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 <email address hidden>[27811]: ERROR nova.api.openstack.extensions MigrationNotFound: Migration 15 could not be found.
Jul 05 20:55:53.751640 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 <email address hidden>[27811]: ERROR nova.api.openstack.extensions

https://github.com/openstack/nova/blob/56cd608d3a199dcb02ac2ae071ff3057241259da/nova/compute/api.py#L4301

We need to use the mappings dict passed to external_instance_event to mutate the context while looking up the migration.

[Matt Riedemann (mriedem)]

This may be fixed by this change:

https://review.openstack.org/#/c/445142/

Specifically this part:

https://review.openstack.org/#/c/445142/10/nova/compute/api.py@4263

I'll add that as a dependency to the devstack 'fleetify' change to test it out:

https://review.openstack.org/#/c/477556/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/445142
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b8be61eb39dc9f605ad853e5697a8f4bf73b025b
Submitter: Jenkins
Branch: master

commit b8be61eb39dc9f605ad853e5697a8f4bf73b025b
Author: Matthew Booth <email address hidden>
Date: Mon Mar 13 18:28:17 2017 +0000

    Fix and optimize external_events for multiple cells

    server_external_events was previously making an api db query and a
    cell db query for every instance reference. We improve this by
    making exactly 1 api db query to fetch all instance mappings, and then
    1 cell db query per cell to fetch all relevant instances from that
    cell. Further, it wasn't properly handling the case where events
    were delivered in one request for multiple instances across cells,
    which this also fixes.

    We also document an obtuse edge condition in
    ComputeAPI.external_instance_event which will cause the current code
    to break when we support migration between cells.

    Note this includes a tweak to the SingleCellSimple fixture to mock out
    the new InstanceMappingList method we use, as well as a fix to the other
    InstanceMapping mock, which was returning mappings with bogus instance
    uuids. This patch relies on the results of those being realistic and
    thus requires those changes.

    Closes-Bug: #1702959

    Co-Authored-By: Dan Smith <email address hidden>

    Change-Id: If59453f1899e99040c554bcb9ad54c8a506adc56

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.0.0.0b3 development milestone.