OpenStack Compute (Nova)

nova API fails to run with Postgres: "UPDATE/SHARE cannot be applied to the nullable side of an outer join"

Reported by Dan Prince on 2012-03-06
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
High
Derek Higgins

Bug Description

Recent Nova API Essex builds (in the past week) no longer run with Postgres:

2012-03-03 00:40:08 DEBUG nova.api.openstack.common [req-8dfe6fa9-c1a3-4826-9f17-e7b351c7a255 admin admin] Generated BUILD from vm_state=building task_state=scheduling. from (pid=10309) status_from_state /usr/lib/python2.7/dist-packages/nova/api/openstack/common.py:96
2012-03-03 00:40:08 ERROR nova.api.openstack [req-8dfe6fa9-c1a3-4826-9f17-e7b351c7a255 admin admin] Caught error: (NotSupportedError) SELECT FOR UPDATE/SHARE cannot be applied to the nullable side of an outer join
 'SELECT compute_nodes.created_at AS compute_nodes_created_at, compute_nodes.updated_at AS compute_nodes_updated_at, compute_nodes.deleted_at AS compute_nodes_deleted_at, compute_nodes.deleted AS compute_nodes_deleted, compute_nodes.id AS compute_nodes_id, compute_nodes.service_id AS compute_nodes_service_id, compute_nodes.vcpus AS compute_nodes_vcpus, compute_nodes.memory_mb AS compute_nodes_memory_mb, compute_nodes.local_gb AS compute_nodes_local_gb, compute_nodes.vcpus_used AS compute_nodes_vcpus_used, compute_nodes.memory_mb_used AS compute_nodes_memory_mb_used, compute_nodes.local_gb_used AS compute_nodes_local_gb_used, compute_nodes.hypervisor_type AS compute_nodes_hypervisor_type, compute_nodes.hypervisor_version AS compute_nodes_hypervisor_version, compute_nodes.hypervisor_hostname AS compute_nodes_hypervisor_hostname, compute_nodes.free_ram_mb AS compute_nodes_free_ram_mb, compute_nodes.free_disk_gb AS compute_nodes_free_disk_gb, compute_nodes.current_workload AS compute_nodes_current_workload, compute_nodes.running_vms AS compute_nodes_running_vms, compute_nodes.cpu_info AS compute_nodes_cpu_info, compute_nodes.disk_available_least AS compute_nodes_disk_available_least, services_1.created_at AS services_1_created_at, services_1.updated_at AS services_1_updated_at, services_1.deleted_at AS services_1_deleted_at, services_1.deleted AS services_1_deleted, services_1.id AS services_1_id, services_1.host AS services_1_host, services_1."binary" AS services_1_binary, services_1.topic AS services_1_topic, services_1.report_count AS services_1_report_count, services_1.disabled AS services_1_disabled, services_1.availability_zone AS services_1_availability_zone \nFROM services, compute_nodes LEFT OUTER JOIN services AS services_1 ON compute_nodes.service_id = services_1.id AND compute_nodes.deleted = %(deleted_1)s \nWHERE services.host IS NULL AND compute_nodes.deleted = %(deleted_2)s \n LIMIT 1 OFFSET 0 FOR UPDATE' {'deleted_2': False, 'deleted_1': False}
(nova.api.openstack): TRACE: Traceback (most recent call last):
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 41, in __call__
(nova.api.openstack): TRACE: return req.get_response(self.application)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1053, in get_response
(nova.api.openstack): TRACE: application, catch_exc_info=False)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1022, in call_application
(nova.api.openstack): TRACE: app_iter = application(self.environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE: return resp(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE: return resp(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE: return resp(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.7/routes/middleware.py", line 131, in __call__
(nova.api.openstack): TRACE: response = self.app(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE: return resp(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
(nova.api.openstack): TRACE: resp = self.call_func(req, *args, **self.kwargs)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 208, in call_func
(nova.api.openstack): TRACE: return self.func(req, *args, **kwargs)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 803, in __call__
(nova.api.openstack): TRACE: content_type, body, accept)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 878, in _process_stack
(nova.api.openstack): TRACE: request, action_args)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 773, in post_process_extensions
(nova.api.openstack): TRACE: **action_args)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/extended_server_attributes.py", line 80, in show
(nova.api.openstack): TRACE: self._extend_server(context, resp_obj.obj['server'], instance)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/extended_server_attributes.py", line 57, in _extend_server
(nova.api.openstack): TRACE: server[key] = self._get_hypervisor_hostname(context, instance)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/extended_server_attributes.py", line 48, in _get_hypervisor_hostname
(nova.api.openstack): TRACE: compute_node = db.compute_node_get_by_host(context, instance["host"])
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 206, in compute_node_get_by_host
(nova.api.openstack): TRACE: return IMPL.compute_node_get_by_host(context, host)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 534, in compute_node_get_by_host
(nova.api.openstack): TRACE: return node.first()
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 1652, in first
(nova.api.openstack): TRACE: ret = list(self[0:1])
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 1560, in __getitem__
(nova.api.openstack): TRACE: return list(res)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 1721, in __iter__
(nova.api.openstack): TRACE: return self._execute_and_instances(context)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 1726, in _execute_and_instances
(nova.api.openstack): TRACE: mapper=self._mapper_zero_or_none())
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 724, in execute
(nova.api.openstack): TRACE: clause, params or {})
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1191, in execute
(nova.api.openstack): TRACE: params)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1271, in _execute_clauseelement
(nova.api.openstack): TRACE: return self.__execute_context(context)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1302, in __execute_context
(nova.api.openstack): TRACE: context.parameters[0], context=context)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1401, in _cursor_execute
(nova.api.openstack): TRACE: context)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1394, in _cursor_execute
(nova.api.openstack): TRACE: context)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 299, in do_execute
(nova.api.openstack): TRACE: cursor.execute(statement, parameters)
(nova.api.openstack): TRACE: NotSupportedError: (NotSupportedError) SELECT FOR UPDATE/SHARE cannot be applied to the nullable side of an outer join
(nova.api.openstack): TRACE: 'SELECT compute_nodes.created_at AS compute_nodes_created_at, compute_nodes.updated_at AS compute_nodes_updated_at, compute_nodes.deleted_at AS compute_nodes_deleted_at, compute_nodes.deleted AS compute_nodes_deleted, compute_nodes.id AS compute_nodes_id, compute_nodes.service_id AS compute_nodes_service_id, compute_nodes.vcpus AS compute_nodes_vcpus, compute_nodes.memory_mb AS compute_nodes_memory_mb, compute_nodes.local_gb AS compute_nodes_local_gb, compute_nodes.vcpus_used AS compute_nodes_vcpus_used, compute_nodes.memory_mb_used AS compute_nodes_memory_mb_used, compute_nodes.local_gb_used AS compute_nodes_local_gb_used, compute_nodes.hypervisor_type AS compute_nodes_hypervisor_type, compute_nodes.hypervisor_version AS compute_nodes_hypervisor_version, compute_nodes.hypervisor_hostname AS compute_nodes_hypervisor_hostname, compute_nodes.free_ram_mb AS compute_nodes_free_ram_mb, compute_nodes.free_disk_gb AS compute_nodes_free_disk_gb, compute_nodes.current_workload AS compute_nodes_current_workload, compute_nodes.running_vms AS compute_nodes_running_vms, compute_nodes.cpu_info AS compute_nodes_cpu_info, compute_nodes.disk_available_least AS compute_nodes_disk_available_least, services_1.created_at AS services_1_created_at, services_1.updated_at AS services_1_updated_at, services_1.deleted_at AS services_1_deleted_at, services_1.deleted AS services_1_deleted, services_1.id AS services_1_id, services_1.host AS services_1_host, services_1."binary" AS services_1_binary, services_1.topic AS services_1_topic, services_1.report_count AS services_1_report_count, services_1.disabled AS services_1_disabled, services_1.availability_zone AS services_1_availability_zone \nFROM services, compute_nodes LEFT OUTER JOIN services AS services_1 ON compute_nodes.service_id = services_1.id AND compute_nodes.deleted = %(deleted_1)s \nWHERE services.host IS NULL AND compute_nodes.deleted = %(deleted_2)s \n LIMIT 1 OFFSET 0 FOR UPDATE' {'deleted_2': False, 'deleted_1': False}
(nova.api.openstack): TRACE:
2012-03-03 00:40:08 INFO nova.api.openstack [req-8dfe6fa9-c1a3-4826-9f17-e7b351c7a255 admin admin] http://172.19.0.3:8774/v1.1/admin/servers/069425e1-6764-4aae-ad42-8de24185f352 returned with HTTP 500

Dan Prince (dan-prince) on 2012-03-06
Changed in nova:
importance: Undecided → High
Derek Higgins (derekh) on 2012-03-06
Changed in nova:
assignee: nobody → Derek Higgins (derekh)
Thierry Carrez (ttx) on 2012-03-08
Changed in nova:
milestone: none → essex-rc1
status: New → In Progress
Derek Higgins (derekh) wrote :

The patch I'm submitting removes the lock mode, I don't think an update cursor is required where this function is used in _get_hypervisor_hostname

Reviewed: https://review.openstack.org/5205
Committed: http://github.com/openstack/nova/commit/5e7fd2584ad8b20635d284bbe448e45a59c37792
Submitter: Jenkins
Branch: master

commit 5e7fd2584ad8b20635d284bbe448e45a59c37792
Author: Derek Higgins <email address hidden>
Date: Sat Mar 10 22:49:29 2012 +0000

    Remove update lockmode from compute_node_get_by_host

    Fixes bug #948066

    This commit removes the usage of an update cursor so postgres
    doesn't raise the error below
    SELECT FOR UPDATE/SHARE cannot be applied to the nullable side of
    an outer join

    Change-Id: Ia1ede22df38d5f6a24372e0ad8ec25151ad195c7

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2012-03-20
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2012-04-05
Changed in nova:
milestone: essex-rc1 → 2012.1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers