| 2013-05-23 09:48:46 |
David McNally |
description |
We are seeing the following error message in the nova-compute logs:
Instance <ID> found in the hypervisor, but not in the database
Upon investigation we found that the instance is running on the hypervisor and correctly entered in the database.
I've tracked this down to a function in the compute manager _get_instances_on_driver. This method is called periodically from
_running_deleted_instances (and possibly from other places). _get_instances_on_driver calls instance_get_all_by_filters and, in this case, the filters are set to retrieve deleted instances on the current host. The list of instances retrieved is compared to the list of instances currently running on the hypervisor. Any instances on the host but not listed in the return from the db query are logged in the above error format, however this means instances which are perfectly healthy are getting logged simply because they are not set to deleted.
This results an error message for every instance on the host being logged each time the periodic task happens. I think we need to remove the log message in this case as it can't know what filters have been specified and so can't provide useful information and just causes noise for anyone monitoring errors. |
We are seeing the following error message in the nova-compute logs:
Instance <ID> found in the hypervisor, but not in the database
Upon investigation we found that the instance is running on the hypervisor and correctly entered in the database.
I've tracked this down to a function in the compute manager _get_instances_on_driver. This method is called periodically from
_running_deleted_instances (and possibly from other places). _get_instances_on_driver calls instance_get_all_by_filters and, in this case, the filters are set to retrieve deleted instances on the current host. The list of instances retrieved is compared to the list of instances currently running on the hypervisor. Any instances on the host but not listed in the return from the db query are logged in the above error format, however this means instances which are perfectly healthy are getting logged simply because they are not set to deleted.
This results in an error message for every instance on the host being logged each time the periodic task happens. I think we need to remove the log message in this case as it can't know what filters have been specified and so can't provide useful information and just causes noise for anyone monitoring errors. |
|
