instance is not destroyed on source host after a successful evacuate

Bug #1518200 reported by lvdongbing
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
lvdongbing
Liberty
Fix Released
Medium
Sylvain Bauza

Bug Description

After evacuate an instance to a new host successfully, then start the old host's nova-compute, bug the old instance not be destroyed as expected.
See following code:
https://github.com/openstack/nova/blob/stable/liberty/nova/compute/manager.py#L817
nova-compute read migration record from db to get the evacuated instance and then destroy it. It filters migration with status 'accepted'.
https://github.com/openstack/nova/blob/stable/liberty/nova/compute/manager.py#L2715
After successfully evacuate instance, status of migration will change to 'done' from 'accepted'
So, I think we should modify 'accepted' to 'done' when filter migration record.

lvdongbing (dbcocle)
Changed in nova:
assignee: nobody → lvdongbing (dbcocle)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/250101

Changed in nova:
status: New → In Progress
Changed in nova:
importance: Undecided → Medium
tags: added: compute rebuild
summary: - instance not be destroyed after evacuate
+ instance is not destroyed on source host after a successful evacuate
Changed in nova:
assignee: lvdongbing (dbcocle) → Sylvain Bauza (sylvain-bauza)
Matt Riedemann (mriedem)
tags: added: liberty-backport-potential
Revision history for this message
Roman Dobosz (roman-dobosz) wrote :

the solution will (probably) fix this bug: https://bugs.launchpad.net/nova/+bug/1532657

Matt Riedemann (mriedem)
Changed in nova:
assignee: Sylvain Bauza (sylvain-bauza) → lvdongbing (dbcocle)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/250101
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a79ac83bae4e673a416cca4b86a3f47f4f25b0aa
Submitter: Jenkins
Branch: master

commit a79ac83bae4e673a416cca4b86a3f47f4f25b0aa
Author: lvdongbing <email address hidden>
Date: Tue Dec 1 04:33:56 2015 -0500

    Fix instance not destroyed after successful evacuation

    After complete evacuate action, the status of migration will change
    to 'done' from 'accepted', so the _destroy_evacuated_instances action
    in old nova-compute should consider the status 'done' to filter
    migration list when they're back up.

    Closes-bug: #1518200
    Co-Authored-By: Sylvain Bauza <email address hidden>

    Change-Id: Ie4825dcaa778d229922c963a96590ad5a2604257

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/269185

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/liberty)

Reviewed: https://review.openstack.org/269185
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d1f371560272b1afb4236ce6a7fd156e64cae44d
Submitter: Jenkins
Branch: stable/liberty

commit d1f371560272b1afb4236ce6a7fd156e64cae44d
Author: lvdongbing <email address hidden>
Date: Tue Dec 1 04:33:56 2015 -0500

    Fix instance not destroyed after successful evacuation

    After complete evacuate action, the status of migration will change
    to 'done' from 'accepted', so the _destroy_evacuated_instances action
    in old nova-compute should consider the status 'done' to filter
    migration list when they're back up.

    Closes-bug: #1518200
    Co-Authored-By: Sylvain Bauza <email address hidden>

    Change-Id: Ie4825dcaa778d229922c963a96590ad5a2604257
    (cherry picked from commit a79ac83bae4e673a416cca4b86a3f47f4f25b0aa)

Revision history for this message
Sean McCully (sean-mccully) wrote :
Download full text (7.1 KiB)

This change raises a psycopg2.ProgrammingError with postgres and psycopg2.

Traceback and Error
==========================
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/eventlet/hubs/poll.py", line 115, in wait
    listener.cb(fileno)
  File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
    result = function(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 671, in run_service
    service.start()
  File "/usr/lib/python2.7/site-packages/nova/service.py", line 164, in start
    self.manager.init_host()
  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1309, in init_host
    self._destroy_evacuated_instances(context)
  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 820, in _destroy_evacuated_instances
    evacuations = objects.MigrationList.get_by_filters(context, filters)
  File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 169, in wrapper
    args, kwargs)
  File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 229, in object_class_action
    args, kwargs)
  File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 237, in object_class_action_versions
    args=args, kwargs=kwargs)
  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in call
    retry=self.retry)
  File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
    timeout=timeout, retry=retry)
  File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 466, in send
    retry=retry)
  File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 457, in _send
    raise result
RemoteError: Remote error: DBError (psycopg2.ProgrammingError) operator does not exist: character varying = text[]
LINE 3: ...HERE migrations.deleted = 0 AND migrations.status = ARRAY['a...
                                                             ^
HINT: No operator matches the given name and argument type(s). You might need to add explicit type casts.
 [SQL: 'SELECT migrations.created_at AS migrations_created_at, migrations.updated_at AS migrations_updated_at, migrations.deleted_at AS migrations_deleted_at, migrations.deleted AS migrations_deleted, migrations.id AS migrations_id, migrations.source_compute AS migrations_source_compute, migrations.dest_compute AS migrations_dest_compute, migrations.source_node AS migrations_source_node, migrations.dest_node AS migrations_dest_node, migrations.dest_host AS migrations_dest_host, migrations.old_instance_type_id AS migrations_old_instance_type_id, migrations.new_instance_type_id AS migrations_new_instance_type_id, migrations.instance_uuid AS migrations_instance_uuid, migrations.status AS migrations_status, migrations.migration_type AS migrations_migration_type, migrations.hidden AS migrations_hidden \nFROM migrations \nWHERE migrations.deleted = %(deleted_1)s AND migrations.status = %(status_1)s AND migrations.source_compute = %(source_compute_1)s AND migrations.migration_type = %(migration_type_1)s'] [parameters: {'deleted_1': 0, 'migration_type_1': u'evacu...

Read more...

Revision history for this message
melanie witt (melwitt) wrote :

Hi Sean,

I've opened a new bug based on your comment:

https://bugs.launchpad.net/nova/+bug/1543382

Matt Riedemann (mriedem)
tags: added: evacuate in-stable-liberty
removed: liberty-backport-potential
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.