LiveMigration DBError: ProgrammingError: can't adapt type 'Instance'

[Tom Patzig]

Postgres as Database + latest stable Kilo; When doing live migrate with attached volume (rally test boot_server_attach_created_volume_and_live_migrate):

----8<----

ERROR oslo_db.sqlalchemy.exc_filters [req-76f003ad-8b12-4e85-902a-a52581fd5611 c2190adfee124fefb139254ce3df27dc 0b707847d6254b5b9e956e40ae16deca - - -] DBAPIError exception wrapped from (ProgrammingError) can't adapt type 'Instance' 'SELECT block_device_mapping.created_at AS block_device_mapping_created_at, block_device_mapping.updated_at AS block_device_mapping_updated_at, block_device_mapping.deleted_at AS block_device_mapping_deleted_at, block_device_mapping.deleted AS block_device_mapping_deleted, block_device_mapping.id AS block_device_mapping_id, block_device_mapping.instance_uuid AS block_device_mapping_instance_uuid, block_device_mapping.source_type AS block_device_mapping_source_type, block_device_mapping.destination_type AS block_device_mapping_destination_type, block_device_mapping.guest_format AS block_device_mapping_guest_format, block_device_mapping.device_type AS block_device_mapping_device_type, block_device_mapping.disk_bus AS block_device_mapping_disk_bus, block_device_mapping.boot_index AS block_device_mapping_boot_index, block_device_mapping.device_name AS block_device_mapping_device_name, block_device_mapping.delete_on_termination AS block_device_mapping_delete_on_termination, block_device_mapping.snapshot_id AS block_device_mapping_snapshot_id, block_device_mapping.volume_id AS block_device_mapping_volume_id, block_device_mapping.volume_size AS block_device_mapping_volume_size, block_device_mapping.image_id AS block_device_mapping_image_id, block_device_mapping.no_device AS block_device_mapping_no_device, block_device_mapping.connection_info AS block_device_mapping_connection_info \nFROM block_device_mapping \nWHERE block_device_mapping.deleted = %(deleted_1)s AND block_device_mapping.volume_id = %(volume_id_1)s \n LIMIT %(param_1)s' {'param_1': 1, 'volume_id_1':
[...]

TRACE oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
TRACE oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1063, in _execute_context
TRACE oslo_db.sqlalchemy.exc_filters context)
TRACE oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 442, in do_execute
TRACE oslo_db.sqlalchemy.exc_filters cursor.execute(statement, parameters)
TRACE oslo_db.sqlalchemy.exc_filters ProgrammingError: can't adapt type 'Instance'

---->8----

In nova-compute.log you find the corresponding error:

-----8<----

TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] Traceback (most recent call last):
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5714, in _live_migration
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] dom, finish_event)
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5651, in _live_migration_monitor
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] migrate_data)
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] payload)
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] six.reraise(self.type_, self.value, self.tb)
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] return f(self, context, *args, **kw)
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 361, in decorated_function
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] kwargs['instance'], e, sys.exc_info())
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] six.reraise(self.type_, self.value, self.tb)
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 349, in decorated_function
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] return function(self, context, *args, **kwargs)
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5564, in _rollback_live_migration
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] context, instance, bdm.volume_id, dest)
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] File "/usr/lib/python2.7/site-packages/nova/compute/rpcapi.py", line 715, in remove_volume_connection
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] instance=instance, volume_id=volume_id)
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] retry=self.retry)
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] timeout=timeout, retry=retry)
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] retry=retry)
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 341, in _send
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] raise result
TRACE nova.virt.libvirt.driver [instance: 70d591fe-cdd8-46c2-bc7c-35ef733c2e1b] RemoteError: Remote error: RemoteError Remote error: DBError (ProgrammingError) can't adapt type 'Instance' 'SELECT block_device_mapping.created_at AS block_device_mapping_created_at, block_device_mapping.updated_at AS block_device_mapping_updated_at, block_device_mapping.deleted_at AS block_device_mapping_deleted_at, block_device_mapping.deleted AS block_device_mapping_deleted, block_device_mapping.id AS block_device_mapping_id, block_device_mapping.instance_uuid AS block_device_mapping_instance_uuid, block_device_mapping.source_type AS block_device_mapping_source_type, block_device_mapping.destination_type AS block_device_mapping_destination_type, block_device_mapping.guest_format AS block_device_mapping_guest_format, block_device_mapping.device_type AS block_device_mapping_device_type, block_device_mapping.disk_bus AS block_device_mapping_disk_bus, block_device_mapping.boot_index AS block_device_mapping_boot_index, block_device_mapping.device_name AS block_device_mapping_device_name, block_device_mapping.delete_on_termination AS block_device_mapping_delete_on_termination, block_device_mapping.snapshot_id AS block_device_mapping_snapshot_id, block_device_mapping.volume_id AS block_device_mapping_volume_id, block_device_mapping.volume_size AS block_device_mapping_volume_size, block_device_mapping.image_id AS block_device_mapping_image_id, block_device_mapping.no_device AS block_device_mapping_no_device, block_device_mapping.connection_info AS block_device_mapping_connection_info \nFROM block_device_mapping \nWHERE block_device_mapping.deleted = %(deleted_1)s AND block_device_mapping.volume_id = %(volume_id_1)s \n LIMIT %(param_1)s' {'param_1': 1, 'volume_id_1':

------->8------

[Matt Riedemann (mriedem)]

What does "can't adapt type 'Instance'" mean in postgres?

[Matt Riedemann (mriedem)]

http://stackoverflow.com/questions/2149515/how-to-fix-cant-adapt-error-when-saving-binary-data-using-python-psycopg2

http://stackoverflow.com/questions/4042783/cant-adapt-type-error-on-a-foreign-key-attribute-with-sqlalchemy-postgresql

What version of sqlalchemy are you running? What versions of postgresql/psycopg2?

[Tom Patzig (tom-patzig)]

The following versions are used:

python-SQLAlchemy-0.9.9-1.2
python-oslo.db-1.7.1-1.2
python-nova-2015.1.3~a0~dev1-9.1
python-psycopg2-2.6-7.1
postgresql93-9.3.6-5.2

[Chris Friesen (cbf123)]

For what it's worth, I just hit what seems to be the same error message, but it turned out to be due to a bug in some instrumentation I had added in qemu. qemu was failing to start, resulting in a log of type:

DBError (ProgrammingError) can't adapt type 'Instance'

as per the above description.

[Markus Zoeller (markus_z) (mzoeller)]

The requested information from comment #2 got provided in comment #3,
so I switch it back to "New".

[Sean Dague (sdague)]

We really need the nova-api or nova-conductor logs here for the root failure here.

[Pushkar Umaranikar (pushkar-umaranikar)]

This bug lacks the necessary information to effectively reproduce and fix it, therefore it has been closed. Feel free to reopen the bug by providing the requested information and set the bug status back to ''New''.