LiveMigration DBError: ProgrammingError: can't adapt type 'Instance'

Bug #1516671 reported by Tom Patzig
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Invalid
Low
Unassigned

Bug Description

Postgres as Database + latest stable Kilo; When doing live migrate with attached volume (rally test boot_server_attach_created_volume_and_live_migrate):

----8<----

ERROR oslo_db.sqlalchemy.exc_filters [req-76f003ad-8b12-4e85-902a-a52581fd5611 c2190adfee124fefb139254ce3df27dc 0b707847d6254b5b9e956e40ae16deca - - -] DBAPIError exception wrapped from (ProgrammingError) can't adapt type 'Instance' 'SELECT block_device_mapping.created_at AS block_device_mapping_created_at, block_device_mapping.updated_at AS block_device_mapping_updated_at, block_device_mapping.deleted_at AS block_device_mapping_deleted_at, block_device_mapping.deleted AS block_device_mapping_deleted, block_device_mapping.id AS block_device_mapping_id, block_device_mapping.instance_uuid AS block_device_mapping_instance_uuid, block_device_mapping.source_type AS block_device_mapping_source_type, block_device_mapping.destination_type AS block_device_mapping_destination_type, block_device_mapping.guest_format AS block_device_mapping_guest_format, block_device_mapping.device_type AS block_device_mapping_device_type, block_device_mapping.disk_bus AS block_device_mapping_disk_bus, block_device_mapping.boot_index AS block_device_mapping_boot_index, block_device_mapping.device_name AS block_device_mapping_device_name, block_device_mapping.delete_on_termination AS block_device_mapping_delete_on_termination, block_device_mapping.snapshot_id AS block_device_mapping_snapshot_id, block_device_mapping.volume_id AS block_device_mapping_volume_id, block_device_mapping.volume_size AS block_device_mapping_volume_size, block_device_mapping.image_id AS block_device_mapping_image_id, block_device_mapping.no_device AS block_device_mapping_no_device, block_device_mapping.connection_info AS block_device_mapping_connection_info \nFROM block_device_mapping \nWHERE block_device_mapping.deleted = %(deleted_1)s AND block_device_mapping.volume_id = %(volume_id_1)s \n LIMIT %(param_1)s' {'param_1': 1, 'volume_id_1':
[...]

TRACE oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
TRACE oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1063, in _execute_context
TRACE oslo_db.sqlalchemy.exc_filters context)
TRACE oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 442, in do_execute
TRACE oslo_db.sqlalchemy.exc_filters cursor.execute(statement, parameters)
TRACE oslo_db.sqlalchemy.exc_filters ProgrammingError: can't adapt type 'Instance'

---->8----

In nova-compute.log you find the corresponding error:

-----8<----

TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] Traceback (most recent call last):
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5714, in _live_migration
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] dom, finish_event)
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5651, in _live_migration_monitor
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] migrate_data)
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] payload)
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] six.reraise(self.type_, self.value, self.tb)
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] return f(self, context, *args, **kw)
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 361, in decorated_function
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] kwargs['instance'], e, sys.exc_info())
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] six.reraise(self.type_, self.value, self.tb)
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 349, in decorated_function
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] return function(self, context, *args, **kwargs)
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5564, in _rollback_live_migration
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] context, instance, bdm.volume_id, dest)
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] File "/usr/lib/python2.7/site-packages/nova/compute/rpcapi.py", line 715, in remove_volume_connection
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] instance=instance, volume_id=volume_id)
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] retry=self.retry)
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] timeout=timeout, retry=retry)
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] retry=retry)
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 341, in _send
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] raise result
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] RemoteError: Remote error: RemoteError Remote error: DBError (ProgrammingError) can't adapt type 'Instance' 'SELECT block_device_mapping.created_at AS block_device_mapping_created_at, block_device_mapping.updated_at AS block_device_mapping_updated_at, block_device_mapping.deleted_at AS block_device_mapping_deleted_at, block_device_mapping.deleted AS block_device_mapping_deleted, block_device_mapping.id AS block_device_mapping_id, block_device_mapping.instance_uuid AS block_device_mapping_instance_uuid, block_device_mapping.source_type AS block_device_mapping_source_type, block_device_mapping.destination_type AS block_device_mapping_destination_type, block_device_mapping.guest_format AS block_device_mapping_guest_format, block_device_mapping.device_type AS block_device_mapping_device_type, block_device_mapping.disk_bus AS block_device_mapping_disk_bus, block_device_mapping.boot_index AS block_device_mapping_boot_index, block_device_mapping.device_name AS block_device_mapping_device_name, block_device_mapping.delete_on_termination AS block_device_mapping_delete_on_termination, block_device_mapping.snapshot_id AS block_device_mapping_snapshot_id, block_device_mapping.volume_id AS block_device_mapping_volume_id, block_device_mapping.volume_size AS block_device_mapping_volume_size, block_device_mapping.image_id AS block_device_mapping_image_id, block_device_mapping.no_device AS block_device_mapping_no_device, block_device_mapping.connection_info AS block_device_mapping_connection_info \nFROM block_device_mapping \nWHERE block_device_mapping.deleted = %(deleted_1)s AND block_device_mapping.volume_id = %(volume_id_1)s \n LIMIT %(param_1)s' {'param_1': 1, 'volume_id_1':

------->8------

Tags: api db
tags: added: db
Revision history for this message
Matt Riedemann (mriedem) wrote :

What does "can't adapt type 'Instance'" mean in postgres?

Revision history for this message
Matt Riedemann (mriedem) wrote :
Changed in nova:
status: New → Incomplete
Revision history for this message
Tom Patzig (tom-patzig) wrote :

The following versions are used:

python-SQLAlchemy-0.9.9-1.2
python-oslo.db-1.7.1-1.2
python-nova-2015.1.3~a0~dev1-9.1
python-psycopg2-2.6-7.1
postgresql93-9.3.6-5.2

Revision history for this message
Chris Friesen (cbf123) wrote :

For what it's worth, I just hit what seems to be the same error message, but it turned out to be due to a bug in some instrumentation I had added in qemu. qemu was failing to start, resulting in a log of type:

DBError (ProgrammingError) can't adapt type 'Instance'

as per the above description.

Revision history for this message
Markus Zoeller (markus_z) (mzoeller) wrote :

The requested information from comment #2 got provided in comment #3,
so I switch it back to "New".

Changed in nova:
status: Incomplete → New
Revision history for this message
Sean Dague (sdague) wrote :

We really need the nova-api or nova-conductor logs here for the root failure here.

Changed in nova:
status: New → Incomplete
importance: Undecided → Low
Revision history for this message
Pushkar Umaranikar (pushkar-umaranikar) wrote :

This bug lacks the necessary information to effectively reproduce and fix it, therefore it has been closed. Feel free to reopen the bug by providing the requested information and set the bug status back to ''New''.

Changed in nova:
status: Incomplete → Invalid
tags: added: api
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.