I see the following stack trace in the logs of a compute node:
2011-06-28 10:54:40,698 AUDIT nova [-] Starting compute node (version 2011.3-workspace:tarmac-2011062519
3807-d47px27v157scd8p)
2011-06-28 10:54:40,862 DEBUG nova.utils [-] backend <module 'nova.db.sqlalchemy.api' from '/usr/lib/pym
odules/python2.7/nova/db/sqlalchemy/api.pyc'> from (pid=12238) __get_backend /usr/lib/pymodules/python2.
7/nova/utils.py:406
2011-06-28 10:54:40,958 CRITICAL nova [-] (OperationalError) (1054, "Unknown column 'instances.image_ref
' in 'field list'") 'SELECT instances.created_at AS instances_created_at, instances.updated_at AS instan
ces_updated_at, instances.deleted_at AS instances_deleted_at, instances.deleted AS instances_deleted, in
stances.id AS instances_id, instances.admin_pass AS instances_admin_pass, instances.user_id AS instances
_user_id, instances.project_id AS instances_project_id, instances.image_ref AS instances_image_ref, inst
ances.kernel_id AS instances_kernel_id, instances.ramdisk_id AS instances_ramdisk_id, instances.launch_i
ndex AS instances_launch_index, instances.key_name AS instances_key_name, instances.key_data AS instance
s_key_data, instances.state AS instances_state, instances.state_description AS instances_state_descripti
on, instances.memory_mb AS instances_memory_mb, instances.vcpus AS instances_vcpus, instances.local_gb A
S instances_local_gb, instances.hostname AS instances_hostname, instances.host AS instances_host, instan
ces.instance_type_id AS instances_instance_type_id, instances.user_data AS instances_user_data, instance
s.reservation_id AS instances_reservation_id, instances.mac_address AS instances_mac_address, instances.
scheduled_at AS instances_scheduled_at, instances.launched_at AS instances_launched_at, instances.termin
ated_at AS instances_terminated_at, instances.availability_zone AS instances_availability_zone, instances.display_name AS instances_display_name, instances.display_description AS instances_display_description, instances.launched_on AS instances_launched_on, instances.locked AS instances_locked, instances.os_type AS instances_os_type, instances.architecture AS instances_architecture, instances.vm_mode AS instances_vm_mode, instances.uuid AS instances_uuid, instance_metadata_1.created_at AS instance_metadata_1_created_at, instance_metadata_1.updated_at AS instance_metadata_1_updated_at, instance_metadata_1.deleted_at AS instance_metadata_1_deleted_at, instance_metadata_1.deleted AS instance_metadata_1_deleted, instance_metadata_1.id AS instance_metadata_1_id, instance_metadata_1.`key` AS instance_metadata_1_key, instance_metadata_1.value AS instance_metadata_1_value, instance_metadata_1.instance_id AS instance_metadata_1_instance_id, security_groups_1.created_at AS security_groups_1_created_at, security_groups_1.updated_at AS security_groups_1_updated_at, security_groups_1.deleted_at AS security_groups_1_deleted_at, security_groups_1.deleted AS security_groups_1_deleted, security_groups_1.id AS security_groups_1_id, security_groups_1.name AS security_groups_1_name, security_groups_1.description AS security_groups_1_description, security_groups_1.user_id AS security_groups_1_user_id, security_groups_1.project_id AS security_groups_1_project_id, instance_types_1.created_at AS instance_types_1_created_at, instance_types_1.updated_at AS instance_types_1_updated_at, instance_types_1.deleted_at AS instance_types_1_deleted_at, instance_types_1.deleted AS instance_types_1_deleted, instance_types_1.id AS instance_types_1_id, instance_types_1.name AS instance_types_1_name, instance_types_1.memory_mb AS instance_types_1_memory_mb, instance_types_1.vcpus AS instance_types_1_vcpus, instance_types_1.local_gb AS instance_types_1_local_gb, instance_types_1.flavorid AS instance_types_1_flavorid, instance_types_1.swap AS instance_types_1_swap, instance_types_1.rxtx_quota AS instance_types_1_rxtx_quota, instance_types_1.rxtx_cap AS instance_types_1_rxtx_cap, networks_1.created_at AS networks_1_created_at, networks_1.updated_at AS networks_1_updated_at, networks_1.deleted_at AS networks_1_deleted_at, networks_1.deleted AS networks_1_deleted, networks_1.id AS networks_1_id, networks_1.label AS networks_1_label, networks_1.injected AS networks_1_injected, networks_1.cidr AS networks_1_cidr, networks_1.cidr_v6 AS networks_1_cidr_v6, networks_1.gateway_v6 AS networks_1_gateway_v6, networks_1.netmask_v6 AS networks_1_netmask_v6, networks_1.netmask AS networks_1_netmask, networks_1.bridge AS networks_1_bridge, networks_1.gateway AS networks_1_gateway, networks_1.broadcast AS networks_1_broadcast, networks_1.dns AS networks_1_dns, networks_1.vlan AS networks_1_vlan, networks_1.vpn_public_address AS networks_1_vpn_public_address, networks_1.vpn_public_port AS networks_1_vpn_public_port, networks_1.vpn_private_address AS networks_1_vpn_private_address, networks_1.dhcp_start AS networks_1_dhcp_start, networks_1.project_id AS networks_1_project_id, networks_1.host AS networks_1_host, fixed_ips_1.created_at AS fixed_ips_1_created_at, fixed_ips_1.updated_at AS fixed_ips_1_updated_at, fixed_ips_1.deleted_at AS fixed_ips_1_deleted_at, fixed_ips_1.deleted AS fixed_ips_1_deleted, fixed_ips_1.id AS fixed_ips_1_id, fixed_ips_1.address AS fixed_ips_1_address, fixed_ips_1.network_id AS fixed_ips_1_network_id, fixed_ips_1.instance_id AS fixed_ips_1_instance_id, fixed_ips_1.allocated AS fixed_ips_1_allocated, fixed_ips_1.leased AS fixed_ips_1_leased, fixed_ips_1.reserved AS fixed_ips_1_reserved, floating_ips_1.created_at AS floating_ips_1_created_at, floating_ips_1.updated_at AS floating_ips_1_updated_at, floating_ips_1.deleted_at AS floating_ips_1_deleted_at, floating_ips_1.deleted AS floating_ips_1_deleted, floating_ips_1.id AS floating_ips_1_id, floating_ips_1.address AS floating_ips_1_address, floating_ips_1.fixed_ip_id AS floating_ips_1_fixed_ip_id, floating_ips_1.project_id AS floating_ips_1_project_id, floating_ips_1.host AS floating_ips_1_host, floating_ips_1.auto_assigned AS floating_ips_1_auto_assigned \nFROM instances LEFT OUTER JOIN instance_metadata AS instance_metadata_1 ON instance_metadata_1.instance_id = instances.id AND instance_metadata_1.deleted = %s LEFT OUTER JOIN security_group_instance_association AS security_group_instance_association_1 ON security_group_instance_association_1.instance_id = instances.id AND instances.deleted = %s LEFT OUTER JOIN security_groups AS security_groups_1 ON security_groups_1.id = security_group_instance_association_1.security_group_id AND security_group_instance_association_1.deleted = %s AND security_groups_1.deleted = %s LEFT OUTER JOIN instance_types AS instance_types_1 ON instances.instance_type_id = instance_types_1.id LEFT OUTER JOIN fixed_ips AS fixed_ips_1 ON fixed_ips_1.instance_id = instances.id AND fixed_ips_1.deleted = %s LEFT OUTER JOIN networks AS networks_1 ON networks_1.id = fixed_ips_1.network_id LEFT OUTER JOIN floating_ips AS floating_ips_1 ON floating_ips_1.fixed_ip_id = fixed_ips_1.id AND floating_ips_1.deleted = %s \nWHERE instances.host = %s AND instances.deleted = %s' (False, False, False, False, False, False, 'compute1', False)
(nova): TRACE: Traceback (most recent call last):
(nova): TRACE: File "/usr/bin/nova-compute", line 48, in <module>
(nova): TRACE: service.serve()
(nova): TRACE: File "/usr/lib/pymodules/python2.7/nova/service.py", line 299, in serve
(nova): TRACE: x.start()
(nova): TRACE: File "/usr/lib/pymodules/python2.7/nova/service.py", line 77, in start
(nova): TRACE: self.manager.init_host()
(nova): TRACE: File "/usr/lib/pymodules/python2.7/nova/compute/manager.py", line 141, in init_host
(nova): TRACE: self.driver.init_host(host=self.host)
(nova): TRACE: File "/usr/lib/pymodules/python2.7/nova/virt/libvirt/connection.py", line 172, in init_host
(nova): TRACE: for instance in db.instance_get_all_by_host(ctxt, host):
(nova): TRACE: File "/usr/lib/pymodules/python2.7/nova/db/api.py", line 449, in instance_get_all_by_host
(nova): TRACE: return IMPL.instance_get_all_by_host(context, host)
(nova): TRACE: File "/usr/lib/pymodules/python2.7/nova/db/sqlalchemy/api.py", line 97, in wrapper
(nova): TRACE: return f(*args, **kwargs)
(nova): TRACE: File "/usr/lib/pymodules/python2.7/nova/db/sqlalchemy/api.py", line 943, in instance_get_all_by_host
(nova): TRACE: filter_by(deleted=can_read_deleted(context)).\
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 1492, in all
(nova): TRACE: return list(self)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 1604, in __iter__
(nova): TRACE: return self._execute_and_instances(context)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 1609, in _execute_and_instances
(nova): TRACE: mapper=self._mapper_zero_or_none())
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 701, in execute
(nova): TRACE: clause, params or {})
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1194, in execute
(nova): TRACE: params)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1274, in _execute_clauseelement
(nova): TRACE: return self.__execute_context(context)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1305, in __execute_context
(nova): TRACE: context.parameters[0], context=context)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1404, in _cursor_execute
(nova): TRACE: context)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1397, in _cursor_execute
(nova): TRACE: context)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 299, in do_execute
(nova): TRACE: cursor.execute(statement, parameters)
(nova): TRACE: File "/usr/lib/pymodules/python2.7/MySQLdb/cursors.py", line 166, in execute
(nova): TRACE: self.errorhandler(self, exc, value)
(nova): TRACE: File "/usr/lib/pymodules/python2.7/MySQLdb/connections.py", line 35, in defaulterrorhandler
(nova): TRACE: raise errorclass, errorvalue
(nova): TRACE: OperationalError: (OperationalError) (1054, "Unknown column 'instances.image_ref' in 'field list'") 'SELECT instances.created_at AS instances_created_at, instances.updated_at AS instances_updated_at, instances.deleted_at AS instances_deleted_at, instances.deleted AS instances_deleted, instances.id AS instances_id, instances.admin_pass AS instances_admin_pass, instances.user_id AS instances_user_id, instances.project_id AS instances_project_id, instances.image_ref AS instances_image_ref, instances.kernel_id AS instances_kernel_id, instances.ramdisk_id AS instances_ramdisk_id, instances.launch_index AS instances_launch_index, instances.key_name AS instances_key_name, instances.key_data AS instances_key_data, instances.state AS instances_state, instances.state_description AS instances_state_description, instances.memory_mb AS instances_memory_mb, instances.vcpus AS instances_vcpus, instances.local_gb AS instances_local_gb, instances.hostname AS instances_hostname, instances.host AS instances_host, instances.instance_type_id AS instances_instance_type_id, instances.user_data AS instances_user_data, instances.reservation_id AS instances_reservation_id, instances.mac_address AS instances_mac_address, instances.scheduled_at AS instances_scheduled_at, instances.launched_at AS instances_launched_at, instances.terminated_at AS instances_terminated_at, instances.availability_zone AS instances_availability_zone, instances.display_name AS instances_display_name, instances.display_description AS instances_display_description, instances.launched_on AS instances_launched_on, instances.locked AS instances_locked, instances.os_type AS instances_os_type, instances.architecture AS instances_architecture, instances.vm_mode AS instances_vm_mode, instances.uuid AS instances_uuid, instance_metadata_1.created_at AS instance_metadata_1_created_at, instance_metadata_1.updated_at AS instance_metadata_1_updated_at, instance_metadata_1.deleted_at AS instance_metadata_1_deleted_at, instance_metadata_1.deleted AS instance_metadata_1_deleted, instance_metadata_1.id AS instance_metadata_1_id, instance_metadata_1.`key` AS instance_metadata_1_key, instance_metadata_1.value AS instance_metadata_1_value, instance_metadata_1.instance_id AS instance_metadata_1_instance_id, security_groups_1.created_at AS security_groups_1_created_at, security_groups_1.updated_at AS security_groups_1_updated_at, security_groups_1.deleted_at AS security_groups_1_deleted_at, security_groups_1.deleted AS security_groups_1_deleted, security_groups_1.id AS security_groups_1_id, security_groups_1.name AS security_groups_1_name, security_groups_1.description AS security_groups_1_description, security_groups_1.user_id AS security_groups_1_user_id, security_groups_1.project_id AS security_groups_1_project_id, instance_types_1.created_at AS instance_types_1_created_at, instance_types_1.updated_at AS instance_types_1_updated_at, instance_types_1.deleted_at AS instance_types_1_deleted_at, instance_types_1.deleted AS instance_types_1_deleted, instance_types_1.id AS instance_types_1_id, instance_types_1.name AS instance_types_1_name, instance_types_1.memory_mb AS instance_types_1_memory_mb, instance_types_1.vcpus AS instance_types_1_vcpus, instance_types_1.local_gb AS instance_types_1_local_gb, instance_types_1.flavorid AS instance_types_1_flavorid, instance_types_1.swap AS instance_types_1_swap, instance_types_1.rxtx_quota AS instance_types_1_rxtx_quota, instance_types_1.rxtx_cap AS instance_types_1_rxtx_cap, networks_1.created_at AS networks_1_created_at, networks_1.updated_at AS networks_1_updated_at, networks_1.deleted_at AS networks_1_deleted_at, networks_1.deleted AS networks_1_deleted, networks_1.id AS networks_1_id, networks_1.label AS networks_1_label, networks_1.injected AS networks_1_injected, networks_1.cidr AS networks_1_cidr, networks_1.cidr_v6 AS networks_1_cidr_v6, networks_1.gateway_v6 AS networks_1_gateway_v6, networks_1.netmask_v6 AS networks_1_netmask_v6, networks_1.netmask AS networks_1_netmask, networks_1.bridge AS networks_1_bridge, networks_1.gateway AS networks_1_gateway, networks_1.broadcast AS networks_1_broadcast, networks_1.dns AS networks_1_dns, networks_1.vlan AS networks_1_vlan, networks_1.vpn_public_address AS networks_1_vpn_public_address, networks_1.vpn_public_port AS networks_1_vpn_public_port, networks_1.vpn_private_address AS networks_1_vpn_private_address, networks_1.dhcp_start AS networks_1_dhcp_start, networks_1.project_id AS networks_1_project_id, networks_1.host AS networks_1_host, fixed_ips_1.created_at AS fixed_ips_1_created_at, fixed_ips_1.updated_at AS fixed_ips_1_updated_at, fixed_ips_1.deleted_at AS fixed_ips_1_deleted_at, fixed_ips_1.deleted AS fixed_ips_1_deleted, fixed_ips_1.id AS fixed_ips_1_id, fixed_ips_1.address AS fixed_ips_1_address, fixed_ips_1.network_id AS fixed_ips_1_network_id, fixed_ips_1.instance_id AS fixed_ips_1_instance_id, fixed_ips_1.allocated AS fixed_ips_1_allocated, fixed_ips_1.leased AS fixed_ips_1_leased, fixed_ips_1.reserved AS fixed_ips_1_reserved, floating_ips_1.created_at AS floating_ips_1_created_at, floating_ips_1.updated_at AS floating_ips_1_updated_at, floating_ips_1.deleted_at AS floating_ips_1_deleted_at, floating_ips_1.deleted AS floating_ips_1_deleted, floating_ips_1.id AS floating_ips_1_id, floating_ips_1.address AS floating_ips_1_address, floating_ips_1.fixed_ip_id AS floating_ips_1_fixed_ip_id, floating_ips_1.project_id AS floating_ips_1_project_id, floating_ips_1.host AS floating_ips_1_host, floating_ips_1.auto_assigned AS floating_ips_1_auto_assigned \nFROM instances LEFT OUTER JOIN instance_metadata AS instance_metadata_1 ON instance_metadata_1.instance_id = instances.id AND instance_metadata_1.deleted = %s LEFT OUTER JOIN security_group_instance_association AS security_group_instance_association_1 ON security_group_instance_association_1.instance_id = instances.id AND instances.deleted = %s LEFT OUTER JOIN security_groups AS security_groups_1 ON security_groups_1.id = security_group_instance_association_1.security_group_id AND security_group_instance_association_1.deleted = %s AND security_groups_1.deleted = %s LEFT OUTER JOIN instance_types AS instance_types_1 ON instances.instance_type_id = instance_types_1.id LEFT OUTER JOIN fixed_ips AS fixed_ips_1 ON fixed_ips_1.instance_id = instances.id AND fixed_ips_1.deleted = %s LEFT OUTER JOIN networks AS networks_1 ON networks_1.id = fixed_ips_1.network_id LEFT OUTER JOIN floating_ips AS floating_ips_1 ON floating_ips_1.fixed_ip_id = fixed_ips_1.id AND floating_ips_1.deleted = %s \nWHERE instances.host = %s AND instances.deleted = %s' (False, False, False, False, False, False, 'compute1', False)
What could be causing this?
Are your database migrations up to date ?
The image_ref column was added in migration 21.
try doing a 'nova-manage db sync' to upgrade the db schema and see if the problem goes away ... if you get any output from the command, the database was out of date.