during the upgrade from Kilo to Mitaka through Liberty live-migration for old instances does not work
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Fix Released
|
Critical
|
Ilya Kharin |
Bug Description
We have an issue during the upgrade of entire environment from Kilo to Mitaka with combined scenario (Mitaka controllers with Liberty upgrade levels + Liberty compute nodes). Live-migration respond with 500 for old instances created in Kilo release.
Steps to reproduce:
1. 7.0 environment:
3xController nodes
3xCeph-OSD nodes
2xCompute nodes
2. Upgrade the Fuel Master node to 8.0 and then to 9.0
3. Upgrade Controllers:
octane upgrade-env
octane upgrade-node --isolated
octane upgrade-db
octane upgrade-ceph
octane upgrade-control
4. Upgrade one Compute node:
octane upgrade-node
Expected results:
All instances are live-migrated for the compute node, node is re-provisioned and re-deployed in Mitaka
Actual results:
Live migration of the first instances fails with the next error:
root@node-5:~# nova live-migration 90156720-
ERROR (ClientException): Unexpected API Error. Please report this at http://
<class 'MigrationError
Action list for the instance:
root@node-5:~# nova instance-action 90156720-
+--
| Property | Value |
+--
| action | live-migration |
| events | [{u'event': u'compute_
| | u'finish_time': u'2016-
| | u'result': u'Error', |
| | u'start_time': u'2016-
| | u'traceback': u' File "/usr/lib/
| | return function(self, context, *args, **kwargs) |
| | File "/usr/lib/
| | kwargs[
| | File "/usr/lib/
| | six.reraise(
| | File "/usr/lib/
| | return function(self, context, *args, **kwargs) |
| | File "/usr/lib/
| | ctxt, instance, refresh_
| | File "/usr/lib/
| | context, instance, self.volume_api, self.driver) |
| | File "/usr/lib/
| | block_device_
| | File "/usr/lib/
| | ret_val = method(obj, context, *args, **kwargs) |
| | File "/usr/lib/
| | connector) |
| | File "/usr/lib/
| | six.reraise(
| | File "/usr/lib/
| | res = method(self, ctx, volume_id, *args, **kwargs) |
| | File "/usr/lib/
| | exc.code if hasattr(exc, \'code\') else None)}) |
| | File "/usr/lib/
| | six.reraise(
| | File "/usr/lib/
| | context)
| | File "/usr/lib/
| | {\'connector\': connector}
| | File "/usr/lib/
| | return self.api.
| | File "/usr/lib/
| | return self._cs_
| | File "/usr/lib/
| | return self.request(url, method, **kwargs) |
| | File "/usr/lib/
| | raise exceptions.
| | '}, |
| | {u'event': u'compute_
| | u'finish_time': u'2016-
| | u'result': u'Error', |
| | u'start_time': u'2016-
| | return function(self, context, *args, **kwargs) |
| | File "/usr/lib/
| | kwargs[
| | File "/usr/lib/
| | six.reraise(
| | File "/usr/lib/
| | return function(self, context, *args, **kwargs) |
| | File "/usr/lib/
| | dest_check_data) |
| | File "/usr/lib/
| | dest_check_
| | File "/usr/lib/
| | retry=self.retry) |
| | File "/usr/lib/
| | timeout=timeout, retry=retry) |
| | File "/usr/lib/
| | retry=retry) |
| | File "/usr/lib/
| | result = self._waiter.
| | File "/usr/lib/
| | message = self.waiters.
| | File "/usr/lib/
| | \'to message ID %s\' % msg_id) |
| | '}] |
| instance_uuid | 90156720-
| message | Error |
| project_id | c9e7f2d26f0c4e7
| request_id | req-d5970b1a-
| start_time | 2016-09-
| user_id | caf71692b020490
+--
root@node-5:~# cinder service-list
+---
| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+---
| cinder-backup | node-2.
| cinder-backup | node-4.
| cinder-backup | node-5.
| cinder-backup | rbd:volumes | nova | enabled | down | 2016-09-
| cinder-scheduler | node-2.
| cinder-scheduler | node-4.
| cinder-scheduler | node-5.
| cinder-scheduler | rbd:volumes | nova | enabled | down | 2016-09-
| cinder-volume | rbd:volumes | nova | enabled | down | 2016-09-
| cinder-volume | rbd:volumes@
+---
./node-
./node-
./node-
./node-
./node-
./node-
./node-
./node-
./node-
./node-
./node-
./node-
./node-
./node-
./node-
./node-
./node-
./node-
./node-
./node-
./node-
./node-
./node-
./node-
./node-
./node-
./node-
./node-
./node-
./node-
./node-
./node-
./node-
./node-
./node-
Changed in fuel: | |
status: | New → Confirmed |
importance: | Undecided → Critical |
milestone: | none → 9.1 |
assignee: | nobody → Ilya Kharin (akscram) |
description: | updated |
Changed in fuel: | |
status: | Confirmed → In Progress |
tags: | added: team-upgrades |
Changed in fuel: | |
status: | In Progress → Fix Committed |
We have to perform update_host for old volumes using `cinder-manage update_host`.