OpenStack Compute (nova)

can't resize after live migrate(block_migrate)

Bug #1240317 reported by jiangguoliang
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Won't Fix
Medium
Unassigned

Bug Description

when i try to resize my instance alter live migrate(block_migrate).i found some error message in nova-compute.log on my compute node.
error message:
2013-10-15 18:52:09.276 ERROR nova.compute.manager [req-8b4330c7-1ea6-404a-ad0d-4f064e6b9643 None None] [instance: 28b509bb-dfe9-4793-a9f8-b121ab16aa6c] t3.uuzu.idc is not a valid node managed by this compute host.. Setting instance vm_state to ERROR
2013-10-15 18:52:09.445 ERROR nova.openstack.common.rpc.amqp [req-8b4330c7-1ea6-404a-ad0d-4f064e6b9643 None None] Exception during message handling
2013-10-15 18:52:09.445 17864 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-10-15 18:52:09.445 17864 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 430, in _process_data
2013-10-15 18:52:09.445 17864 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)
2013-10-15 18:52:09.445 17864 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 133, in dispatch
2013-10-15 18:52:09.445 17864 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-10-15 18:52:09.445 17864 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
2013-10-15 18:52:09.445 17864 TRACE nova.openstack.common.rpc.amqp temp_level, payload)
2013-10-15 18:52:09.445 17864 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-10-15 18:52:09.445 17864 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-10-15 18:52:09.445 17864 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped
2013-10-15 18:52:09.445 17864 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
2013-10-15 18:52:09.445 17864 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in decorated_function
2013-10-15 18:52:09.445 17864 TRACE nova.openstack.common.rpc.amqp function(self, context, *args, **kwargs)
2013-10-15 18:52:09.445 17864 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in decorated_function
2013-10-15 18:52:09.445 17864 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info())
2013-10-15 18:52:09.445 17864 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-10-15 18:52:09.445 17864 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-10-15 18:52:09.445 17864 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in decorated_function
2013-10-15 18:52:09.445 17864 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-10-15 18:52:09.445 17864 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2050, in confirm_resize
2013-10-15 18:52:09.445 17864 TRACE nova.openstack.common.rpc.amqp rt = self._get_resource_tracker(migration['source_node'])
2013-10-15 18:52:09.445 17864 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 361, in _get_resource_tracker
2013-10-15 18:52:09.445 17864 TRACE nova.openstack.common.rpc.amqp raise exception.NovaException(msg)
2013-10-15 18:52:09.445 17864 TRACE nova.openstack.common.rpc.amqp NovaException: t3.uuzu.idc is not a valid node managed by this compute host.
2013-10-15 18:52:09.445 17864 TRACE nova.openstack.common.rpc.amqp

I found the instance is on the new host t3.uuzu.idc. but the status is ERROR.
Version is Grizzly.

jiangguoliang (jglyq1982)
Changed in nova:
assignee: nobody → jiangguoliang (jglyq1982)
Revision history for this message
Matt Riedemann (mriedem) wrote :

Are there any errors in the nova scheduler log? Are there any different errors listed in the instance fault property for the instance?

Russell Bryant (russellb)
tags: added: compute
Revision history for this message
melanie witt (melwitt) wrote :

Asking for more information from user.

Changed in nova:
status: New → Incomplete
Revision history for this message
haruka tanizawa (h-tanizawa) wrote :

Any update on this?

Revision history for this message
Sean Dague (sdague) wrote :

Long incomplete bug, please reopen if you can provide the new info.

Changed in nova:
assignee: jiangguoliang (jglyq1982) → nobody
status: Incomplete → Invalid
Revision history for this message
Rey Aram A. Alcantara (reyaram-alcantara) wrote :
Download full text (8.6 KiB)

2014-10-01 22:50:22.612 24766 ERROR nova.compute.manager [req-c7c8f643-6b3a-4ca2-99c6-be252dc12b87 fba8a3983484473ebb5caf6af441c282 04d61c2227bc448397fcbf035dd76890] [instance: fbefc220-0ef0-4c4d-a06d-31709e022e25] Setting instance vm_state to ERROR
2014-10-01 22:50:22.612 24766 TRACE nova.compute.manager [instance: fbefc220-0ef0-4c4d-a06d-31709e022e25] Traceback (most recent call last):
2014-10-01 22:50:22.612 24766 TRACE nova.compute.manager [instance: fbefc220-0ef0-4c4d-a06d-31709e022e25] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5538, in _error_out_instance_on_exception
2014-10-01 22:50:22.612 24766 TRACE nova.compute.manager [instance: fbefc220-0ef0-4c4d-a06d-31709e022e25] yield
2014-10-01 22:50:22.612 24766 TRACE nova.compute.manager [instance: fbefc220-0ef0-4c4d-a06d-31709e022e25] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3117, in _confirm_resize
2014-10-01 22:50:22.612 24766 TRACE nova.compute.manager [instance: fbefc220-0ef0-4c4d-a06d-31709e022e25] rt = self._get_resource_tracker(migration.source_node)
2014-10-01 22:50:22.612 24766 TRACE nova.compute.manager [instance: fbefc220-0ef0-4c4d-a06d-31709e022e25] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 613, in _get_resource_tracker
2014-10-01 22:50:22.612 24766 TRACE nova.compute.manager [instance: fbefc220-0ef0-4c4d-a06d-31709e022e25] "compute host.") % nodename)
2014-10-01 22:50:22.612 24766 TRACE nova.compute.manager [instance: fbefc220-0ef0-4c4d-a06d-31709e022e25] NovaException: compute2 is not a valid node managed by this compute host.
2014-10-01 22:50:22.612 24766 TRACE nova.compute.manager [instance: fbefc220-0ef0-4c4d-a06d-31709e022e25]
2014-10-01 22:50:22.747 24766 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: compute2 is not a valid node managed by this compute host.
2014-10-01 22:50:22.747 24766 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-10-01 22:50:22.747 24766 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-10-01 22:50:22.747 24766 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-10-01 22:50:22.747 24766 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-10-01 22:50:22.747 24766 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-10-01 22:50:22.747 24766 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-10-01 22:50:22.747 24766 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-10-01 22:50:22.747 24766 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
2014-10-01 22:50:22.747 24766 TRACE oslo.messaging.rpc.dispatcher payload)
2014-10-01 22:50:22.747 24766 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excuti...

Read more...

Changed in nova:
status: Invalid → New
Revision history for this message
Rey Aram A. Alcantara (reyaram-alcantara) wrote :

I forgot to add the destination compute current resource avialability:

2014-10-02 06:51:52.006 4020 AUDIT nova.compute.resource_tracker [-] Updating from migration 269e5088-4570-48f0-a6a6-a914e9b06afa
2014-10-02 06:51:52.007 4020 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 107889
2014-10-02 06:51:52.008 4020 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 107
2014-10-02 06:51:52.008 4020 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 22
2014-10-02 06:51:52.040 4020 INFO nova.compute.resource_tracker [-] Compute_service record updated for compute2:compute2

Here is also the services manage by Nova:

Binary Host Zone Status State Updated_At
nova-consoleauth controller1 internal enabled :-) 2014-10-02 00:24:23
nova-conductor controller1 internal enabled :-) 2014-10-02 00:24:17
nova-cert controller1 internal enabled :-) 2014-10-02 00:24:24
nova-scheduler controller1 internal enabled :-) 2014-10-02 00:24:21
nova-compute compute1 nova enabled :-) 2014-10-02 00:24:22
nova-network compute1 internal enabled :-) 2014-10-02 00:24:21
nova-compute compute2 nova enabled :-) 2014-10-02 00:24:26
nova-network compute2 internal enabled :-) 2014-10-02 00:24:16
nova-compute compute3 nova enabled :-) 2014-10-02 00:24:16
nova-network compute3 internal enabled :-) 2014-10-02 00:24:20
nova-compute compute4 nova enabled :-) 2014-10-02 00:24:25
nova-network compute4 internal enabled :-) 2014-10-02 00:24:19
nova-compute compute5 nova enabled :-) 2014-10-02 00:24:25
nova-network compute5 internal enabled :-) 2014-10-02 00:24:22
nova-compute compute6 nova enabled :-) 2014-10-02 00:24:18
nova-network compute6 internal enabled :-) 2014-10-02 00:24:20
nova-compute compute7 nova enabled :-) 2014-10-02 00:24:24
nova-network compute7 internal enabled :-) 2014-10-02 00:24:25

Revision history for this message
Sylvain Bauza (sylvain-bauza) wrote :

The bug seems to be valid, the driver is incorrectly reporting the node as unavailable

Changed in nova:
status: New → Triaged
importance: Undecided → Medium
Sean Dague (sdague)
summary: - cant resize alter live migrate(block_migrate)
+ can't resize after live migrate(block_migrate)
Changed in nova:
status: Triaged → Confirmed
Timofey Durakov (tdurakov)
Changed in nova:
assignee: nobody → Timofey Durakov (tdurakov)
Pawel Koniszewski (pawel-koniszewski)
tags: added: live-migration
tags: added: resize
tags: added: live-migrate
removed: live-migration
Revision history for this message
Eli Qiao (taget-9) wrote :
Download full text (17.9 KiB)

hi , I 'd like to mark this as incomplete because I can not reproduce it with your step in my environment.

taget@liyong:/opt/stack/nova$ git log
commit 183cd88cb2f9781b53f71b6b161df401c286c9ff
Merge: 3ee98a6 2833f8c
Author: Jenkins <email address hidden>
Date: Sun Jul 26 13:34:34 2015 +0000

    Merge "remove _rescan_iscsi from disconnect_volume_multipath_iscsi"

taget@liyong:/opt/stack$ nova live-migration test1 liyong2 --block-migrate
taget@liyong:/opt/stack$ nova show test1
+--------------------------------------+----------------------------------------------------------------+
| Property | Value |
 +--------------------------------------+----------------------------------------------------------------+
| 192_net network | 192.168.0.32 |
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | liyong2 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | liyong2 |
| OS-EXT-SRV-ATTR:instance_name | instance-00000061 |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2015-08-28T07:00:41.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2015-08-28T07:00:31Z |
| flavor | m1.tiny (1) |
| hostId | 7a2a33df86ec0a0aa08ab1337c6adf00e80573c5cefd403b05a02ceb |
| id | e00e9012-2832-4256-b820-ec5ae4769dd8 |
| image | cirros-0.3.2-x86_64-uec (93169d18-1bb2-45f7-861c-6735ef687db9) |
| key_name | - |
| metadata | {} |
| name | test1 |
| os-extended-volumes:volumes_attached | [] ...

Changed in nova:
status: Confirmed → Incomplete
Paul Murray (pmurray)
tags: added: live-migration
removed: live-migrate
Revision history for this message
Markus Zoeller (markus_z) (mzoeller) wrote :

Cleanup
=======

This bug report has the status "Incomplete" since more than 30 days
and it looks like that there are no open reviews for it. To keep
the bug list sane, I close this bug with "won't fix". This does not
mean that it is not a valid bug report, it's more to acknowledge that
no progress can be expected here anymore. You are still free to push a
new patch for this bug. If you could reproduce it on the current master
code or on a maintained stable branch, please switch it to "Confirmed".

Changed in nova:
status: Incomplete → Won't Fix
assignee: Timofey Durakov (tdurakov) → nobody
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.