After service deleted, the corresponding compute-node can't restart again

Bug #1418298 reported by Alex Xu
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Alex Xu

Bug Description

After remove a stopped service for nova-compute, then I can't restart it again. The error as below

2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/openstack/common/threadgroup.py", line 145, in wait
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup x.wait()
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/openstack/common/threadgroup.py", line 47, in wait
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup return self.thread.wait()
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 173, in wait
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup return self._exit_event.wait()
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup return hubs.get_hub().switch()
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 293, in switch
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup return self.greenlet.switch()
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 212, in main
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup result = function(*args, **kwargs)
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/openstack/common/service.py", line 492, in run_service
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup service.start()
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/service.py", line 181, in start
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup self.manager.pre_start_hook()
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/compute/manager.py", line 1181, in pre_start_hook
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup self.update_available_resource(nova.context.get_admin_context())
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/compute/manager.py", line 6058, in update_available_resource
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup rt.update_available_resource(context)
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/compute/resource_tracker.py", line 342, in update_available_resource
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup return self._update_available_resource(context, resources)
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 431, in inner
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup return f(*args, **kwargs)
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/compute/resource_tracker.py", line 396, in _update_available_resource
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup self._sync_compute_node(context, resources)
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/compute/resource_tracker.py", line 417, in _sync_compute_node
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup self._create(context, resources)
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/compute/resource_tracker.py", line 466, in _create
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup values)
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/conductor/api.py", line 170, in compute_node_create
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup return self._manager.compute_node_create(context, values)
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/conductor/rpcapi.py", line 271, in compute_node_create
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup return cctxt.call(context, 'compute_node_create', values=values)
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 152, in call
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup retry=self.retry)
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup timeout=timeout, retry=retry)
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 408, in send
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup retry=retry)
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 399, in _send
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup raise result

2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup RemoteError: Remote error: DBDuplicateEntry (IntegrityError) (1062, "Duplicate entry 'hp-pc-hp-pc' for key 'uniq_compute_nodes0host0hypervisor_hostname'") 'INSERT INTO compute_nodes (created_at, updated_at, deleted_at, deleted, service_id, host, vcpus, memory_mb, local_gb, vcpus_used, memory_mb_used, local_gb_used, hypervisor_type, hypervisor_version, hypervisor_hostname, free_ram_mb, free_disk_gb, current_workload, running_vms, cpu_info, disk_available_least, host_ip, supported_instances, metrics, pci_stats, extra_resources, stats, numa_topology) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)' (datetime.datetime(2015, 2, 5, 3, 9, 3, 245004), None, None, 0, 6, 'hp-pc', 8, 7920, 45, 0, 512, 0, 'QEMU', 2000000, 'hp-pc', 7408, 45, 0, 0, '{"vendor": "Intel", "model": "Nehalem", "arch": "x86_64", "features": ["pge", "clflush", "sep", "syscall", "vme", "dtes64", "tsc", "vmx", "xtpr", "cmov", "ssse3", "est", "pat", "monitor", "smx", "lm", "msr", "nx", "fxsr", "tm", "sse4.1", "pae", "sse4.2", "acpi", "de", "mmx", "cx8", "mce", "mtrr", "rdtscp", "ht", "pse", "lahf_lm", "pdcm", "mca", "apic", "sse", "ds", "pni", "tm2", "sse2", "ss", "pbe", "fpu", "cx16", "pse36", "ds_cpl", "popcnt"], "topology": {"cores": 4, "threads": 2, "sockets": 1}}', 34, '10.238.154.76', '[["alpha", "qemu", "hvm"], ["armv7l", "qemu", "hvm"], ["cris", "qemu", "hvm"], ["i686", "qemu", "hvm"], ["i686", "kvm", "hvm"], ["lm32", "qemu", "hvm"], ["m68k", "qemu", "hvm"], ["microblaze", "qemu", "hvm"], ["microblazeel", "qemu", "hvm"], ["mips", "qemu", "hvm"], ["mipsel", "qemu", "hvm"], ["mips64", "qemu", "hvm"], ["mips64el", "qemu", "hvm"], ["ppc", "qemu", "hvm"], ["ppc64", "qemu", "hvm"], ["ppcemb", "qemu", "hvm"], ["s390x", "qemu", "hvm"], ["sh4", "qemu", "hvm"], ["sh4eb", "qemu", "hvm"], ["sparc", "qemu", "hvm"], ["sparc64", "qemu", "hvm"], ["unicore32", "qemu", "hvm"], ["x86_64", "qemu", "hvm"], ["x86_64", "kvm", "hvm"], ["xtensa", "qemu", "hvm"], ["xtensaeb", "qemu", "hvm"]]', '[]', '[]', None, '{}', '{"nova_object.version": "1.2", "nova_object.changes": ["cells"], "nova_object.name": "NUMATopology", "nova_object.data": {"cells": [{"nova_object.version": "1.2", "nova_object.changes": ["cpu_usage", "memory_usage", "cpuset", "pinned_cpus", "siblings", "memory", "mempages", "id"], "nova_object.name": "NUMACell", "nova_object.data": {"cpu_usage": 0, "memory_usage": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7], "pinned_cpus": [], "siblings": [[0, 1], [6, 7], [2, 4], [3, 5]], "memory": 7920, "mempages": [], "id": 0}, "nova_object.namespace": "nova"}]}, "nova_object.namespace": "nova"}')
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup [u'Traceback (most recent call last):\n', u' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n incoming.message))\n', u' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', u' File "/opt/stack/nova/nova/conductor/manager.py", line 313, in compute_node_create\n result = self.db.compute_node_create(context, values)\n', u' File "/opt/stack/nova/nova/db/api.py", line 247, in compute_node_create\n return IMPL.compute_node_create(context, values)\n', u' File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 128, in wrapper\n return f(*args, **kwargs)\n', u' File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 599, in compute_node_create\n compute_node_ref.save()\n', u' File "/opt/stack/nova/nova/db/sqlalchemy/models.py", line 82, in save\n super(NovaBase, self).save(session=session)\n', u' File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/models.py", line 48, in save\n session.flush()\n', u' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1919, in flush\n self._flush(objects)\n', u' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2037, in _flush\n transaction.rollback(_capture_exception=True)\n', u' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__\n compat.reraise(exc_type, exc_value, exc_tb)\n', u' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2001, in _flush\n flush_context.execute()\n', u' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 372, in execute\n rec.execute(self)\n', u' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 526, in execute\n uow\n', u' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 65, in save_obj\n mapper, table, insert)\n', u' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 602, in _emit_insert_statements\n execute(statement, params)\n', u' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 729, in execute\n return meth(self, multiparams, params)\n', u' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 322, in _execute_on_connection\n return connection._execute_clauseelement(self, multiparams, params)\n', u' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement\n compiled_sql, distilled_params\n', u' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 958, in _execute_context\n context)\n', u' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1155, in _handle_dbapi_exception\n util.raise_from_cause(newraise, exc_info)\n', u' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause\n reraise(type(exception), exception, tb=exc_tb)\n', u' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 951, in _execute_context\n context)\n', u' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 436, in do_execute\n cursor.execute(statement, parameters)\n', u' File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute\n self.errorhandler(self, exc, value)\n', u' File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler\n raise errorclass, errorvalue\n', u'DBDuplicateEntry: (IntegrityError) (1062, "Duplicate entry \'hp-pc-hp-pc\' for key \'uniq_compute_nodes0host0hypervisor_hostname\'") \'INSERT INTO compute_nodes (created_at, updated_at, deleted_at, deleted, service_id, host, vcpus, memory_mb, local_gb, vcpus_used, memory_mb_used, local_gb_used, hypervisor_type, hypervisor_version, hypervisor_hostname, free_ram_mb, free_disk_gb, current_workload, running_vms, cpu_info, disk_available_least, host_ip, supported_instances, metrics, pci_stats, extra_resources, stats, numa_topology) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)\' (datetime.datetime(2015, 2, 5, 3, 9, 3, 245004), None, None, 0, 6, \'hp-pc\', 8, 7920, 45, 0, 512, 0, \'QEMU\', 2000000, \'hp-pc\', 7408, 45, 0, 0, \'{"vendor": "Intel", "model": "Nehalem", "arch": "x86_64", "features": ["pge", "clflush", "sep", "syscall", "vme", "dtes64", "tsc", "vmx", "xtpr", "cmov", "ssse3", "est", "pat", "monitor", "smx", "lm", "msr", "nx", "fxsr", "tm", "sse4.1", "pae", "sse4.2", "acpi", "de", "mmx", "cx8", "mce", "mtrr", "rdtscp", "ht", "pse", "lahf_lm", "pdcm", "mca", "apic", "sse", "ds", "pni", "tm2", "sse2", "ss", "pbe", "fpu", "cx16", "pse36", "ds_cpl", "popcnt"], "topology": {"cores": 4, "threads": 2, "sockets": 1}}\', 34, \'10.238.154.76\', \'[["alpha", "qemu", "hvm"], ["armv7l", "qemu", "hvm"], ["cris", "qemu", "hvm"], ["i686", "qemu", "hvm"], ["i686", "kvm", "hvm"], ["lm32", "qemu", "hvm"], ["m68k", "qemu", "hvm"], ["microblaze", "qemu", "hvm"], ["microblazeel", "qemu", "hvm"], ["mips", "qemu", "hvm"], ["mipsel", "qemu", "hvm"], ["mips64", "qemu", "hvm"], ["mips64el", "qemu", "hvm"], ["ppc", "qemu", "hvm"], ["ppc64", "qemu", "hvm"], ["ppcemb", "qemu", "hvm"], ["s390x", "qemu", "hvm"], ["sh4", "qemu", "hvm"], ["sh4eb", "qemu", "hvm"], ["sparc", "qemu", "hvm"], ["sparc64", "qemu", "hvm"], ["unicore32", "qemu", "hvm"], ["x86_64", "qemu", "hvm"], ["x86_64", "kvm", "hvm"], ["xtensa", "qemu", "hvm"], ["xtensaeb", "qemu", "hvm"]]\', \'[]\', \'[]\', None, \'{}\', \'{"nova_object.version": "1.2", "nova_object.changes": ["cells"], "nova_object.name": "NUMATopology", "nova_object.data": {"cells": [{"nova_object.version": "1.2", "nova_object.changes": ["cpu_usage", "memory_usage", "cpuset", "pinned_cpus", "siblings", "memory", "mempages", "id"], "nova_object.name": "NUMACell", "nova_object.data": {"cpu_usage": 0, "memory_usage": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7], "pinned_cpus": [], "siblings": [[0, 1], [6, 7], [2, 4], [3, 5]], "memory": 7920, "mempages": [], "id": 0}, "nova_object.namespace": "nova"}]}, "nova_object.namespace": "nova"}\')\n'].
2015-02-05 11:09:03.302 TRACE nova.openstack.common.threadgroup

Alex Xu (xuhj)
Changed in nova:
assignee: nobody → Alex Xu (xuhj)
summary: - after service deleted, the corresponding compute-node can't restart
+ After service deleted, then corresponding compute-node can't restart
again
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/153123

Changed in nova:
status: New → In Progress
Alex Xu (xuhj)
Changed in nova:
importance: Undecided → High
Alex Xu (xuhj)
summary: - After service deleted, then corresponding compute-node can't restart
+ After service deleted, the corresponding compute-node can't restart
again
Changed in nova:
milestone: none → kilo-3
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/153123
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2db4a1accb78536cf0606addf9ca79938b408542
Submitter: Jenkins
Branch: master

commit 2db4a1accb78536cf0606addf9ca79938b408542
Author: He Jie Xu <email address hidden>
Date: Thu Feb 5 10:57:27 2015 +0800

    Update unique constraint of compute_nodes with deleted column

    After a service deleted, the corresponding compute node entry will be
    soft-deleted also. But we can't startup this compute node again, because
    when create new entry for this compute node, the current compute_nodes table's
    unique constraint was violated.

    This patch adds 'deleted' column into unique constraint to fix this problem.

    Change-Id: I71ca360df629a4df71892e8595d3ef1707f8d115
    Closes-Bug: #1418298

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: kilo-3 → 2015.1.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.