On a HA cluster, after restating keepalived , most of the instances gets stuck in Build Launching instances becomes very slow.. It becomes slight better after 20-30 mins Someimes the below trace is also seen in nova- conductor root@cs-scale-3:/var/log/nova# tailf /var/log/nova/nova-conductor.log tailf /var/log/nova/nova-conductor.log 2015-06-22 21:41:34.933 769 ERROR nova.scheduler.utils [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a None] [instance: 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3] Error from last host: cs-scale-6 (node cs-scale-6): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2051, in _do_build_and_run_instance\n filter_properties)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2182, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3 was re-scheduled: Timed out waiting for a reply to message ID afe0230e020e4c2e9d5b6e62d0531407\n'] 2015-06-22 21:41:34.936 769 INFO oslo.messaging._drivers.impl_rabbit [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a ] Connecting to AMQP server on 13.1.0.10:5673 root@cs-scale-2:/var/log/nova# nova show 35d15e01-710c-4f49-b6e6-b99f42578011 +--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Property | Value | +--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | cs-scale-8 | | OS-EXT-SRV-ATTR:hypervisor_hostname | cs-scale-8 | | OS-EXT-SRV-ATTR:instance_name | instance-00001c78 | | OS-EXT-STS:power_state | 0 | | OS-EXT-STS:task_state | block_device_mapping | | OS-EXT-STS:vm_state | error | | OS-SRV-USG:launched_at | - | | OS-SRV-USG:terminated_at | - | | accessIPv4 | | | accessIPv6 | | | config_drive | | | created | 2015-06-23T04:38:42Z | | fault | {"message": "Build of instance 35d15e01-710c-4f49-b6e6-b99f42578011 aborted: Failure prepping block device.", "code": 500, "details": " File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2051, in _do_build_and_run_instance | | | filter_properties) | | | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2150, in _build_and_run_instance | | | 'create.error', fault=e) | | | File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py\", line 82, in __exit__ | | | six.reraise(self.type_, self.value, self.tb) | | | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2123, in _build_and_run_instance | | | block_device_mapping) as resources: | | | File \"/usr/lib/python2.7/contextlib.py\", line 17, in __enter__ | | | return self.gen.next() | | | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2261, in _build_resources | | | reason=msg) | | | ", "created": "2015-06-23T04:58:09Z"} | | flavor | V1 (10) | | hostId | d5a27e5eda6edfaef28db46cb3db4a55228b5e94f4dd18650f075f3f | | id | 35d15e01-710c-4f49-b6e6-b99f42578011 | | image | A1-SNAP1 (3da3258d-c630-4396-aca9-017f7f62d7fb) | | key_name | - | | metadata | {} | | name | VIN-1--35d15e01-710c-4f49-b6e6-b99f42578011 | | os-extended-volumes:volumes_attached | [] | | status | ERROR | | tenant_id | 1657ff89c9c54aa1b42d100a4a663a1f | | updated | 2015-06-23T04:58:09Z | | user_id | a9b90ce5977d44ec8b65ada3d31c4747 | +--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2015-06-22 21:41:34.933 769 ERROR nova.scheduler.utils [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a None] [instance: 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3] Error from last host: cs-scale-6 (node cs-scale-6): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2051, in _do_build_and_run_instance\n filter_properties)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2182, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3 was re-scheduled: Timed out waiting for a reply to message ID afe0230e020e4c2e9d5b6e62d0531407\n'] 2015-06-22 21:41:34.936 769 INFO oslo.messaging._drivers.impl_rabbit [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a ] Connecting to AMQP server on 13.1.0.10:5673