Failed to live-migrate instance in cell with microversion >= 2.34

[Yikun Jiang]

Step 1 create instance in cell1
+--------------------------------------+--------+--------+------------+-------------+---------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------+--------+------------+-------------+---------------------------------+
| 84038890-8d70-45e1-8240-2303f4227e11 | yikun1 | ACTIVE | - | Running | public=2001:db8::a, 172.24.4.13 |
+--------------------------------------+--------+--------+------------+-------------+---------------------------------+

Step 2 live migrate instance
nova live-migration 84038890-8d70-45e1-8240-2303f4227e11

Step 3
The instance will stuck in "MIGRATIING" state.
+--------------------------------------+--------+-----------+------------+-------------+---------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------+-----------+------------+-------------+---------------------------------+
| 84038890-8d70-45e1-8240-2303f4227e11 | yikun1 | MIGRATING | migrating | Running | public=2001:db8::a, 172.24.4.13 |
+--------------------------------------+--------+-----------+------------+-------------+---------------------------------+

It seems we need add @targets_cell decorator for **live_migrate_instance** methods in conductor:
https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L378

ERROR LOG in super conductor:
Exception during message handling: InstanceActionNotFound: Action for request_id req-5aa03558-ae14-458e-9c35-c3d377c7ce45 on instance 84038890-8d70-45e1-8240-2303f4227e11 not found
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming
    res = self.dispatcher.dispatch(message)
  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
    result = func(ctxt, **new_args)
  File "/opt/stack/nova/nova/compute/utils.py", line 875, in decorated_function
    with EventReporter(context, event_name, instance_uuid):
  File "/opt/stack/nova/nova/compute/utils.py", line 846, in __enter__
    self.context, uuid, self.event_name, want_result=False)
  File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
    result = fn(cls, context, *args, **kwargs)
  File "/opt/stack/nova/nova/objects/instance_action.py", line 169, in event_start
    db_event = db.action_event_start(context, values)
  File "/opt/stack/nova/nova/db/api.py", line 1957, in action_event_start
    return IMPL.action_event_start(context, values)
  File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 250, in wrapped
    return f(context, *args, **kwargs)
  File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 6155, in action_event_start
    instance_uuid=values['instance_uuid'])
InstanceActionNotFound: Action for request_id req-5aa03558-ae14-458e-9c35-c3d377c7ce45 on instance 84038890-8d70-45e1-8240-2303f4227e11 not found

[Zhenyu Zheng (zhengzhenyu)]

https://review.openstack.org/#/c/503601

[Matt Riedemann (mriedem)]

This needs more details. When we lookup the instance in the API we target the context to the cell that the instance is in, and then that context is used when the API calls conductor so we should be fine, plus we have live migration CI tests which are not failing, so this seems invalid.

[Matt Riedemann (mriedem)]

Is your super conductor nova.conf configured to hit the [api_database]?

[Yikun Jiang (yikunkero)]

Sorry for the lack of information I given.

In this case, the api cast live migrate rpc to super-conductor,
and the default db of super-conductor is cell0, but instance
action of this instance infomation store in cell1, the super-conductor
wasn't aware of this without targets_cell decorator.

According to implements of bp/cells-aware-api, I found that the rpc cast
from compute to conductor DON'T refresh the transport of conductor rpc
client. So, it will cast to super-condcuotr and ctxt in conductor doesn't
 contain the mq or db connection which refresh in api.

That is,
1. now api just cast message to the **super-conductor**(yep, not cell-conducotr)
2. super-conductor is aware of the instance db_connection depends on
targets_cell decorator.

Unfortunately, live_migrate_intance method does't add this decoractor.
I think, maybe, Dan might omitted this function when add decorators for
every super conductor operation.(https://review.openstack.org/#/c/438022/)

Thus, in order to let super conductor know which db the instance use,
we should add decorator for live_migrate_intance method.

[Yikun Jiang (yikunkero)]

Is your super conductor nova.conf configured to hit the [api_database]?

@matt seems no, my config as below:

### /etc/nova/nova_cell1.conf
[DEFAULT]
transport_url = rabbit://stackrabbit:1@XXXXX:5672/nova_cell1
[database]
connection = mysql+pymysql://root:1@127.0.0.1/nova_cell1?charset=utf8

### /etc/nova/nova.conf
[DEFAULT]
transport_url = rabbit://stackrabbit:1@XXXXX:5672/
[database]
connection = mysql+pymysql://root:1@127.0.0.1/nova_cell0?charset=utf8
[api_database]
connection = mysql+pymysql://root:1@127.0.0.1/nova_api?charset=utf8

### super-conductor process
/usr/bin/python /usr/local/bin/nova-conductor --config-file /etc/nova/nova.conf

### cell-condutor process
/usr/bin/python /usr/local/bin/nova-conductor --config-file /etc/nova/nova_cell1.conf

[Yikun Jiang (yikunkero)]

BTW, Without the @targets_cell decorator:
1. (before casting)API context.db_connection in api is **cell1** db connection.
I print api context before casting:
context.db_connection._root._root_factory.__dict__ ==>'connection': u'mysql+pymysql://root:1@127.0.0.1/nova_cell1

2. (after casting)Super-Conductor context.db_connection is None.

[Matt Riedemann (mriedem)]

The reason the live migration CI job didn't catch this is that it doesn't test microversion >= 2.34 which changes to use the live_migrate_instance method. If the microversion < 2.34 it uses the migrate_server method which has the @targets_cell decorator.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/505285

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/503601
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=062f5b2e876a09119e43c1905f91610cd4e5d015
Submitter: Jenkins
Branch: master

commit 062f5b2e876a09119e43c1905f91610cd4e5d015
Author: Yikun Jiang <email address hidden>
Date: Wed Sep 13 19:35:49 2017 +0800

    Add @targets_cell for live_migrate_instance method in conductor

    With microversion < 2.34, the API casts to the migrate_server
    method in super conductor which targets the context using the
    @targets_cell decorator.

    With microversion >= 2.34, the API casts to the live_migrate_instance
    method in super conductor which does not use the @targets_cell
    decorator, which results in a failure to lookup the instance action
    record when recording the start of the action event with the
    @wrap_instance_event decorator.

    This change simply adds the decorator and provides a test which
    was missing for this before. Note that the live migration CI job
    didn't catch this regression since it only tests up to microversion
    2.26.

    Co-Authored-By: Matt Riedemann <email address hidden>

    Closes-bug: #1716903
    Change-Id: I21d3f3b7589221b7e0a46c332510afc876ca5a79

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/505285
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a9f9e70dd08295618731445f38af2031adc859f1
Submitter: Jenkins
Branch: stable/pike

commit a9f9e70dd08295618731445f38af2031adc859f1
Author: Yikun Jiang <email address hidden>
Date: Wed Sep 13 19:35:49 2017 +0800

    Add @targets_cell for live_migrate_instance method in conductor

    With microversion < 2.34, the API casts to the migrate_server
    method in super conductor which targets the context using the
    @targets_cell decorator.

    With microversion >= 2.34, the API casts to the live_migrate_instance
    method in super conductor which does not use the @targets_cell
    decorator, which results in a failure to lookup the instance action
    record when recording the start of the action event with the
    @wrap_instance_event decorator.

    This change simply adds the decorator and provides a test which
    was missing for this before. Note that the live migration CI job
    didn't catch this regression since it only tests up to microversion
    2.26.

    Co-Authored-By: Matt Riedemann <email address hidden>

    Closes-bug: #1716903
    Change-Id: I21d3f3b7589221b7e0a46c332510afc876ca5a79
    (cherry picked from commit 062f5b2e876a09119e43c1905f91610cd4e5d015)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.0.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.0.0b1 development milestone.