Instance hangs in migrating state after live-migration

Bug #1346865 reported by Tatyanka
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
New
High
Fuel Library (Deprecated)

Bug Description

test http://jenkins-product.srt.mirantis.net:8080/view/5.0_swarm/job/5.0_fuelmain.system_test.centos.thread_1/47/testReport/%28root%29/migrate_vm_backed_with_ceph/migrate_vm_backed_with_ceph/ failed with

http://paste.openstack.org/show/87591/

according to instance hangs in Migration state, and we receive 400 Bad request when try to get details for this instance.
[root@node-2 ~]# nova list
+--------------------------------------+---------------------+-----------+------------+-------------+-------------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+---------------------+-----------+------------+-------------+-------------------------------------+
| 118994db-3ef5-467a-876f-6c3cdffeae41 | test-serv1147266757 | MIGRATING | migrating | Running | novanetwork=10.0.0.5, 10.108.52.191 |
+--------------------------------------+---------------------+-----------+------------+-------------+-------------------------------------+

Attemtps to get server diagnostics finished with
[root@node-2 ~]# nova diagnostics test-serv1147266757

ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-cb48c9ab-eb37-42a7-99c3-b0e46fc2ffdb)

instance reset state push instance to error
[root@node-2 ~]# nova reboot test-serv1147266757
ERROR: Cannot 'reboot' while instance is in task_state migrating (HTTP 409) (Request-ID: req-09a63ecb-b9bc-46e9-afaf-5fd5f1c87bdc)
[root@node-2 ~]# nova reset-state test-serv1147266757
[root@node-2 ~]# nova list
+--------------------------------------+---------------------+--------+------------+-------------+-------------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+---------------------+--------+------------+-------------+-------------------------------------+
| 118994db-3ef5-467a-876f-6c3cdffeae41 | test-serv1147266757 | ERROR | - | Running | novanetwork=10.0.0.5, 10.108.52.191 |
+--------------------------------------+---------------------+--------+------------+-------------+-------------------------------------+
[root@node-2 ~]#

Seems it caused by resseting connections to the rabbitmq
http://paste.openstack.org/show/87593/
errors in rabbit=ERROR REPORT==== 22-Jul-2014::04:39:27 ===
closing AMQP connection <0.11236.0> (10.108.54.4:37774 -> 10.108.54.3:5672):
{heartbeat_timeout,running}

=ERROR REPORT==== 22-Jul-2014::04:39:32 ===
closing AMQP connection <0.11722.0> (10.108.54.2:46623 -> 10.108.54.3:5672):
{heartbeat_timeout,running}

=INFO REPORT==== 22-Jul-2014::04:45:10 ===
accepting AMQP connection <0.21380.0> (10.108.54.2:46695 -> 10.108.54.3:5672)

=ERROR REPORT==== 22-Jul-2014::04:47:33 ===
closing AMQP connection <0.12140.0> (10.108.54.2:46626 -> 10.108.54.3:5672):
{heartbeat_timeout,running}

=INFO REPORT==== 22-Jul-2014::04:49:35 ===
accepting AMQP connection <0.23212.0> (10.108.54.3:38884 -> 10.108.54.3:5672)

=INFO REPORT==== 22-Jul-2014::04:49:40 ===
accepting AMQP connection <0.23247.0> (10.108.54.3:38885 -> 10.108.54.3:5672)

=ERROR REPORT==== 22-Jul-2014::04:52:35 ===
closing AMQP connection <0.23212.0> (10.108.54.3:38884 -> 10.108.54.3:5672):
{heartbeat_timeout,running}

=ERROR REPORT==== 22-Jul-2014::04:52:40 ===
closing AMQP connection <0.23247.0> (10.108.54.3:38885 -> 10.108.54.3:5672):
{heartbeat_timeout,running}
(END)

Seems this is duplicate of https://bugs.launchpad.net/mos/+bug/856764 but we nned confirmation from our best and lovelly library team :)

Tags: library
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.