| 2015-06-23 07:15:00 |
Vinod Nair |
bug |
|
|
added bug |
| 2015-06-23 07:15:12 |
Vinod Nair |
nominated for series |
|
juniperopenstack/trunk |
|
| 2015-06-23 07:15:12 |
Vinod Nair |
bug task added |
|
juniperopenstack/trunk |
|
| 2015-06-23 07:15:12 |
Vinod Nair |
nominated for series |
|
juniperopenstack/r2.20 |
|
| 2015-06-23 07:15:12 |
Vinod Nair |
bug task added |
|
juniperopenstack/r2.20 |
|
| 2015-06-23 07:15:23 |
Vinod Nair |
juniperopenstack/r2.20: milestone |
|
r2.20-fcs |
|
| 2015-06-23 07:15:32 |
Vinod Nair |
juniperopenstack/trunk: milestone |
|
r2.30-fcs |
|
| 2015-06-23 07:15:40 |
Vinod Nair |
juniperopenstack/r2.20: assignee |
|
Jeya ganesh babu (jjeya) |
|
| 2015-06-23 07:15:47 |
Vinod Nair |
juniperopenstack/trunk: assignee |
|
Jeya ganesh babu (jjeya) |
|
| 2015-06-23 07:15:50 |
Vinod Nair |
juniperopenstack/r2.20: importance |
Undecided |
Critical |
|
| 2015-06-23 07:15:52 |
Vinod Nair |
juniperopenstack/trunk: importance |
Undecided |
Critical |
|
| 2015-06-23 07:23:29 |
Vinod Nair |
description |
On a HA cluster, after restating keepalived , most of the instances gets stuck in Build
Launching instances becomes very slow.. It becomes slight better after 20-30 mins
Someimes the below trace is also seen in nova- conductor
root@cs-scale-3:/var/log/nova# tailf /var/log/nova/nova-conductor.log
tailf /var/log/nova/nova-conductor.log
2015-06-22 21:41:34.933 769 ERROR nova.scheduler.utils [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a None] [instance: 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3] Error from last host: cs-scale-6 (node cs-scale-6): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2051, in _do_build_and_run_instance\n filter_properties)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2182, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3 was re-scheduled: Timed out waiting for a reply to message ID afe0230e020e4c2e9d5b6e62d0531407\n']
2015-06-22 21:41:34.936 769 INFO oslo.messaging._drivers.impl_rabbit [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a ] Connecting to AMQP server on 13.1.0.10:5673
root@cs-scale-2:/var/log/nova# nova show 35d15e01-710c-4f49-b6e6-b99f42578011
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | cs-scale-8 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | cs-scale-8 |
| OS-EXT-SRV-ATTR:instance_name | instance-00001c78 |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | block_device_mapping |
| OS-EXT-STS:vm_state | error |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2015-06-23T04:38:42Z |
| fault | {"message": "Build of instance 35d15e01-710c-4f49-b6e6-b99f42578011 aborted: Failure prepping block device.", "code": 500, "details": " File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2051, in _do_build_and_run_instance |
| | filter_properties) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2150, in _build_and_run_instance |
| | 'create.error', fault=e) |
| | File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py\", line 82, in __exit__ |
| | six.reraise(self.type_, self.value, self.tb) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2123, in _build_and_run_instance |
| | block_device_mapping) as resources: |
| | File \"/usr/lib/python2.7/contextlib.py\", line 17, in __enter__ |
| | return self.gen.next() |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2261, in _build_resources |
| | reason=msg) |
| | ", "created": "2015-06-23T04:58:09Z"} |
| flavor | V1 (10) |
| hostId | d5a27e5eda6edfaef28db46cb3db4a55228b5e94f4dd18650f075f3f |
| id | 35d15e01-710c-4f49-b6e6-b99f42578011 |
| image | A1-SNAP1 (3da3258d-c630-4396-aca9-017f7f62d7fb) |
| key_name | - |
| metadata | {} |
| name | VIN-1--35d15e01-710c-4f49-b6e6-b99f42578011 |
| os-extended-volumes:volumes_attached | [] |
| status | ERROR |
| tenant_id | 1657ff89c9c54aa1b42d100a4a663a1f |
| updated | 2015-06-23T04:58:09Z |
| user_id | a9b90ce5977d44ec8b65ada3d31c4747 |
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
2015-06-22 21:41:34.933 769 ERROR nova.scheduler.utils [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a None] [instance: 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3] Error from last host: cs-scale-6 (node cs-scale-6): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2051, in _do_build_and_run_instance\n filter_properties)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2182, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3 was re-scheduled: Timed out waiting for a reply to message ID afe0230e020e4c2e9d5b6e62d0531407\n']
2015-06-22 21:41:34.936 769 INFO oslo.messaging._drivers.impl_rabbit [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a ] Connecting to AMQP server on 13.1.0.10:5673 |
On a HA cluster, after restating keepalived , most of the instances gets stuck in Build
Launching instances becomes very slow.. It becomes slight better after 20-30 mins
Someimes the below trace is also seen in nova- conductor
root@cs-scale-3:/var/log/nova# tailf /var/log/nova/nova-conductor.log
tailf /var/log/nova/nova-conductor.log
2015-06-22 21:41:34.933 769 ERROR nova.scheduler.utils [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a None] [instance: 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3] Error from last host: cs-scale-6 (node cs-scale-6): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2051, in _do_build_and_run_instance\n filter_properties)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2182, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3 was re-scheduled: Timed out waiting for a reply to message ID afe0230e020e4c2e9d5b6e62d0531407\n']
2015-06-22 21:41:34.936 769 INFO oslo.messaging._drivers.impl_rabbit [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a ] Connecting to AMQP server on 13.1.0.10:5673
2015-06-22 21:41:34.933 769 ERROR nova.scheduler.utils [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a None] [instance: 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3] Error from last host: cs-scale-6 (node cs-scale-6): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2051, in _do_build_and_run_instance\n filter_properties)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2182, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3 was re-scheduled: Timed out waiting for a reply to message ID afe0230e020e4c2e9d5b6e62d0531407\n']
2015-06-22 21:41:34.936 769 INFO oslo.messaging._drivers.impl_rabbit [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a ] Connecting to AMQP server on 13.1.0.10:5673 |
|
| 2015-06-23 07:26:12 |
Vinod Nair |
description |
On a HA cluster, after restating keepalived , most of the instances gets stuck in Build
Launching instances becomes very slow.. It becomes slight better after 20-30 mins
Someimes the below trace is also seen in nova- conductor
root@cs-scale-3:/var/log/nova# tailf /var/log/nova/nova-conductor.log
tailf /var/log/nova/nova-conductor.log
2015-06-22 21:41:34.933 769 ERROR nova.scheduler.utils [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a None] [instance: 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3] Error from last host: cs-scale-6 (node cs-scale-6): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2051, in _do_build_and_run_instance\n filter_properties)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2182, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3 was re-scheduled: Timed out waiting for a reply to message ID afe0230e020e4c2e9d5b6e62d0531407\n']
2015-06-22 21:41:34.936 769 INFO oslo.messaging._drivers.impl_rabbit [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a ] Connecting to AMQP server on 13.1.0.10:5673
2015-06-22 21:41:34.933 769 ERROR nova.scheduler.utils [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a None] [instance: 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3] Error from last host: cs-scale-6 (node cs-scale-6): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2051, in _do_build_and_run_instance\n filter_properties)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2182, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3 was re-scheduled: Timed out waiting for a reply to message ID afe0230e020e4c2e9d5b6e62d0531407\n']
2015-06-22 21:41:34.936 769 INFO oslo.messaging._drivers.impl_rabbit [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a ] Connecting to AMQP server on 13.1.0.10:5673 |
On a HA cluster, after restating keepalived , most of the instances gets stuck in Build
Launching instances becomes very slow.. It becomes slight better after 20-30 mins
Someimes the below trace is also seen in nova- conductor
root@cs-scale-3:/var/log/nova# tailf /var/log/nova/nova-conductor.log
tailf /var/log/nova/nova-conductor.log
2015-06-22 21:41:34.933 769 ERROR nova.scheduler.utils [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a None] [instance: 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3] Error from last host: cs-scale-6 (node cs-scale-6): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2051, in _do_build_and_run_instance\n filter_properties)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2182, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3 was re-scheduled: Timed out waiting for a reply to message ID afe0230e020e4c2e9d5b6e62d0531407\n']
2015-06-22 21:41:34.936 769 INFO oslo.messaging._drivers.impl_rabbit [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a ] Connecting to AMQP server on 13.1.0.10:5673
2015-06-22 21:41:34.933 769 ERROR nova.scheduler.utils [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a None] [instance: 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3] Error from last host: cs-scale-6 (node cs-scale-6): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2051, in _do_build_and_run_instance\n filter_properties)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2182, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3 was re-scheduled: Timed out waiting for a reply to message ID afe0230e020e4c2e9d5b6e62d0531407\n']
2015-06-22 21:41:34.936 769 INFO oslo.messaging._drivers.impl_rabbit [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a ] Connecting to AMQP server on 13.1.0.10:5673
root@cs-scale-2:/var/log/nova# nova show eeca56e7-4a13-4787-9820-75f3ea07f24c
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | cs-scale-5 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | cs-scale-5 |
| OS-EXT-SRV-ATTR:instance_name | instance-00001e1f |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | block_device_mapping |
| OS-EXT-STS:vm_state | error |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2015-06-23T06:53:19Z https://bugs.launchpad.net/juniperopenstack/+bug/1467782/+nominate |
| fault | {"message": "Build of instance eeca56e7-4a13-4787-9820-75f3ea07f24c aborted: Failure prepping block device.", "code": 500, "details": " File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2051, in _do_build_and_run_instance |
| | filter_properties) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2150, in _build_and_run_instance |
| | 'create.error', fault=e) |
| | File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py\", line 82, in __exit__ |
| | six.reraise(self.type_, self.value, self.tb) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2123, in _build_and_run_instance |
| | block_device_mapping) as resources: |
| | File \"/usr/lib/python2.7/contextlib.py\", line 17, in __enter__ |
| | return self.gen.next() |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2261, in _build_resources |
| | reason=msg) |
| | ", "created": "2015-06-23T07:12:01Z"} |
| flavor | V1 (10) |
| hostId | b0f49b19c9d1a90b3207e2e3a88b14c36062d4a61e6e92801a3c1262 |
| id | eeca56e7-4a13-4787-9820-75f3ea07f24c |
| image | A1-SNAP1 (3da3258d-c630-4396-aca9-017f7f62d7fb) |
| key_name | - |
| metadata | {} |
| name | VIN2--eeca56e7-4a13-4787-9820-75f3ea07f24c |
| os-extended-volumes:volumes_attached | [] |
| status | ERROR |
| tenant_id | 1657ff89c9c54aa1b42d100a4a663a1f |
| updated | 2015-06-23T07:12:01Z |
| user_id | a9b90ce5977d44ec8b65ada3d31c4747 |
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
root@cs-scale-2:/var/log/nova# |
|
| 2015-06-23 07:27:06 |
Vinod Nair |
description |
On a HA cluster, after restating keepalived , most of the instances gets stuck in Build
Launching instances becomes very slow.. It becomes slight better after 20-30 mins
Someimes the below trace is also seen in nova- conductor
root@cs-scale-3:/var/log/nova# tailf /var/log/nova/nova-conductor.log
tailf /var/log/nova/nova-conductor.log
2015-06-22 21:41:34.933 769 ERROR nova.scheduler.utils [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a None] [instance: 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3] Error from last host: cs-scale-6 (node cs-scale-6): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2051, in _do_build_and_run_instance\n filter_properties)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2182, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3 was re-scheduled: Timed out waiting for a reply to message ID afe0230e020e4c2e9d5b6e62d0531407\n']
2015-06-22 21:41:34.936 769 INFO oslo.messaging._drivers.impl_rabbit [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a ] Connecting to AMQP server on 13.1.0.10:5673
2015-06-22 21:41:34.933 769 ERROR nova.scheduler.utils [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a None] [instance: 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3] Error from last host: cs-scale-6 (node cs-scale-6): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2051, in _do_build_and_run_instance\n filter_properties)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2182, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3 was re-scheduled: Timed out waiting for a reply to message ID afe0230e020e4c2e9d5b6e62d0531407\n']
2015-06-22 21:41:34.936 769 INFO oslo.messaging._drivers.impl_rabbit [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a ] Connecting to AMQP server on 13.1.0.10:5673
root@cs-scale-2:/var/log/nova# nova show eeca56e7-4a13-4787-9820-75f3ea07f24c
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | cs-scale-5 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | cs-scale-5 |
| OS-EXT-SRV-ATTR:instance_name | instance-00001e1f |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | block_device_mapping |
| OS-EXT-STS:vm_state | error |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2015-06-23T06:53:19Z https://bugs.launchpad.net/juniperopenstack/+bug/1467782/+nominate |
| fault | {"message": "Build of instance eeca56e7-4a13-4787-9820-75f3ea07f24c aborted: Failure prepping block device.", "code": 500, "details": " File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2051, in _do_build_and_run_instance |
| | filter_properties) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2150, in _build_and_run_instance |
| | 'create.error', fault=e) |
| | File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py\", line 82, in __exit__ |
| | six.reraise(self.type_, self.value, self.tb) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2123, in _build_and_run_instance |
| | block_device_mapping) as resources: |
| | File \"/usr/lib/python2.7/contextlib.py\", line 17, in __enter__ |
| | return self.gen.next() |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2261, in _build_resources |
| | reason=msg) |
| | ", "created": "2015-06-23T07:12:01Z"} |
| flavor | V1 (10) |
| hostId | b0f49b19c9d1a90b3207e2e3a88b14c36062d4a61e6e92801a3c1262 |
| id | eeca56e7-4a13-4787-9820-75f3ea07f24c |
| image | A1-SNAP1 (3da3258d-c630-4396-aca9-017f7f62d7fb) |
| key_name | - |
| metadata | {} |
| name | VIN2--eeca56e7-4a13-4787-9820-75f3ea07f24c |
| os-extended-volumes:volumes_attached | [] |
| status | ERROR |
| tenant_id | 1657ff89c9c54aa1b42d100a4a663a1f |
| updated | 2015-06-23T07:12:01Z |
| user_id | a9b90ce5977d44ec8b65ada3d31c4747 |
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
root@cs-scale-2:/var/log/nova# |
On a HA cluster, after restating keepalived , most of the instances gets stuck in Build
Launching instances becomes very slow.. It becomes slight better after 20-30 mins
LOGS in : http://cmbu-sv02.englab.juniper.net/pxe/Standard/vin/ha/b1/
version : 2.20 build 59 Juno with patches
Someimes the below trace is also seen in nova- conductor
root@cs-scale-3:/var/log/nova# tailf /var/log/nova/nova-conductor.log
tailf /var/log/nova/nova-conductor.log
2015-06-22 21:41:34.933 769 ERROR nova.scheduler.utils [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a None] [instance: 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3] Error from last host: cs-scale-6 (node cs-scale-6): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2051, in _do_build_and_run_instance\n filter_properties)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2182, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3 was re-scheduled: Timed out waiting for a reply to message ID afe0230e020e4c2e9d5b6e62d0531407\n']
2015-06-22 21:41:34.936 769 INFO oslo.messaging._drivers.impl_rabbit [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a ] Connecting to AMQP server on 13.1.0.10:5673
2015-06-22 21:41:34.933 769 ERROR nova.scheduler.utils [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a None] [instance: 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3] Error from last host: cs-scale-6 (node cs-scale-6): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2051, in _do_build_and_run_instance\n filter_properties)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2182, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3 was re-scheduled: Timed out waiting for a reply to message ID afe0230e020e4c2e9d5b6e62d0531407\n']
2015-06-22 21:41:34.936 769 INFO oslo.messaging._drivers.impl_rabbit [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a ] Connecting to AMQP server on 13.1.0.10:5673
root@cs-scale-2:/var/log/nova# nova show eeca56e7-4a13-4787-9820-75f3ea07f24c
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | cs-scale-5 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | cs-scale-5 |
| OS-EXT-SRV-ATTR:instance_name | instance-00001e1f |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | block_device_mapping |
| OS-EXT-STS:vm_state | error |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2015-06-23T06:53:19Z https://bugs.launchpad.net/juniperopenstack/+bug/1467782/+nominate |
| fault | {"message": "Build of instance eeca56e7-4a13-4787-9820-75f3ea07f24c aborted: Failure prepping block device.", "code": 500, "details": " File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2051, in _do_build_and_run_instance |
| | filter_properties) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2150, in _build_and_run_instance |
| | 'create.error', fault=e) |
| | File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py\", line 82, in __exit__ |
| | six.reraise(self.type_, self.value, self.tb) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2123, in _build_and_run_instance |
| | block_device_mapping) as resources: |
| | File \"/usr/lib/python2.7/contextlib.py\", line 17, in __enter__ |
| | return self.gen.next() |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2261, in _build_resources |
| | reason=msg) |
| | ", "created": "2015-06-23T07:12:01Z"} |
| flavor | V1 (10) |
| hostId | b0f49b19c9d1a90b3207e2e3a88b14c36062d4a61e6e92801a3c1262 |
| id | eeca56e7-4a13-4787-9820-75f3ea07f24c |
| image | A1-SNAP1 (3da3258d-c630-4396-aca9-017f7f62d7fb) |
| key_name | - |
| metadata | {} |
| name | VIN2--eeca56e7-4a13-4787-9820-75f3ea07f24c |
| os-extended-volumes:volumes_attached | [] |
| status | ERROR |
| tenant_id | 1657ff89c9c54aa1b42d100a4a663a1f |
| updated | 2015-06-23T07:12:01Z |
| user_id | a9b90ce5977d44ec8b65ada3d31c4747 |
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
root@cs-scale-2:/var/log/nova# |
|
| 2015-06-23 07:37:39 |
Vinod Nair |
description |
On a HA cluster, after restating keepalived , most of the instances gets stuck in Build
Launching instances becomes very slow.. It becomes slight better after 20-30 mins
LOGS in : http://cmbu-sv02.englab.juniper.net/pxe/Standard/vin/ha/b1/
version : 2.20 build 59 Juno with patches
Someimes the below trace is also seen in nova- conductor
root@cs-scale-3:/var/log/nova# tailf /var/log/nova/nova-conductor.log
tailf /var/log/nova/nova-conductor.log
2015-06-22 21:41:34.933 769 ERROR nova.scheduler.utils [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a None] [instance: 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3] Error from last host: cs-scale-6 (node cs-scale-6): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2051, in _do_build_and_run_instance\n filter_properties)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2182, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3 was re-scheduled: Timed out waiting for a reply to message ID afe0230e020e4c2e9d5b6e62d0531407\n']
2015-06-22 21:41:34.936 769 INFO oslo.messaging._drivers.impl_rabbit [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a ] Connecting to AMQP server on 13.1.0.10:5673
2015-06-22 21:41:34.933 769 ERROR nova.scheduler.utils [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a None] [instance: 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3] Error from last host: cs-scale-6 (node cs-scale-6): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2051, in _do_build_and_run_instance\n filter_properties)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2182, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3 was re-scheduled: Timed out waiting for a reply to message ID afe0230e020e4c2e9d5b6e62d0531407\n']
2015-06-22 21:41:34.936 769 INFO oslo.messaging._drivers.impl_rabbit [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a ] Connecting to AMQP server on 13.1.0.10:5673
root@cs-scale-2:/var/log/nova# nova show eeca56e7-4a13-4787-9820-75f3ea07f24c
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | cs-scale-5 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | cs-scale-5 |
| OS-EXT-SRV-ATTR:instance_name | instance-00001e1f |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | block_device_mapping |
| OS-EXT-STS:vm_state | error |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2015-06-23T06:53:19Z https://bugs.launchpad.net/juniperopenstack/+bug/1467782/+nominate |
| fault | {"message": "Build of instance eeca56e7-4a13-4787-9820-75f3ea07f24c aborted: Failure prepping block device.", "code": 500, "details": " File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2051, in _do_build_and_run_instance |
| | filter_properties) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2150, in _build_and_run_instance |
| | 'create.error', fault=e) |
| | File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py\", line 82, in __exit__ |
| | six.reraise(self.type_, self.value, self.tb) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2123, in _build_and_run_instance |
| | block_device_mapping) as resources: |
| | File \"/usr/lib/python2.7/contextlib.py\", line 17, in __enter__ |
| | return self.gen.next() |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2261, in _build_resources |
| | reason=msg) |
| | ", "created": "2015-06-23T07:12:01Z"} |
| flavor | V1 (10) |
| hostId | b0f49b19c9d1a90b3207e2e3a88b14c36062d4a61e6e92801a3c1262 |
| id | eeca56e7-4a13-4787-9820-75f3ea07f24c |
| image | A1-SNAP1 (3da3258d-c630-4396-aca9-017f7f62d7fb) |
| key_name | - |
| metadata | {} |
| name | VIN2--eeca56e7-4a13-4787-9820-75f3ea07f24c |
| os-extended-volumes:volumes_attached | [] |
| status | ERROR |
| tenant_id | 1657ff89c9c54aa1b42d100a4a663a1f |
| updated | 2015-06-23T07:12:01Z |
| user_id | a9b90ce5977d44ec8b65ada3d31c4747 |
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
root@cs-scale-2:/var/log/nova# |
On a HA cluster, after restating keepalived , most of the instances gets stuck in Build
Launching instances becomes very slow.. It becomes slight better after 20-30 mins
LOGS in : http://cmbu-sv02.englab.juniper.net/pxe/Standard/vin/ha/b1/
version : 2.20 build 59 Juno with patches
Logs: http://cmbu-sv02.englab.juniper.net/pxe/Standard/vin/ha/b1/
Someimes the below trace is also seen in nova- conductor
root@cs-scale-3:/var/log/nova# tailf /var/log/nova/nova-conductor.log
tailf /var/log/nova/nova-conductor.log
2015-06-22 21:41:34.933 769 ERROR nova.scheduler.utils [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a None] [instance: 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3] Error from last host: cs-scale-6 (node cs-scale-6): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2051, in _do_build_and_run_instance\n filter_properties)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2182, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3 was re-scheduled: Timed out waiting for a reply to message ID afe0230e020e4c2e9d5b6e62d0531407\n']
2015-06-22 21:41:34.936 769 INFO oslo.messaging._drivers.impl_rabbit [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a ] Connecting to AMQP server on 13.1.0.10:5673
2015-06-22 21:41:34.933 769 ERROR nova.scheduler.utils [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a None] [instance: 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3] Error from last host: cs-scale-6 (node cs-scale-6): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2051, in _do_build_and_run_instance\n filter_properties)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2182, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 89a31e4e-e52b-4cf8-8a07-ea99b74c61d3 was re-scheduled: Timed out waiting for a reply to message ID afe0230e020e4c2e9d5b6e62d0531407\n']
2015-06-22 21:41:34.936 769 INFO oslo.messaging._drivers.impl_rabbit [req-a6ed435f-b08c-4cdf-9c6f-b39d88e0ac6a ] Connecting to AMQP server on 13.1.0.10:5673
root@cs-scale-2:/var/log/nova# nova show eeca56e7-4a13-4787-9820-75f3ea07f24c
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | cs-scale-5 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | cs-scale-5 |
| OS-EXT-SRV-ATTR:instance_name | instance-00001e1f |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | block_device_mapping |
| OS-EXT-STS:vm_state | error |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2015-06-23T06:53:19Z https://bugs.launchpad.net/juniperopenstack/+bug/1467782/+nominate |
| fault | {"message": "Build of instance eeca56e7-4a13-4787-9820-75f3ea07f24c aborted: Failure prepping block device.", "code": 500, "details": " File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2051, in _do_build_and_run_instance |
| | filter_properties) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2150, in _build_and_run_instance |
| | 'create.error', fault=e) |
| | File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py\", line 82, in __exit__ |
| | six.reraise(self.type_, self.value, self.tb) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2123, in _build_and_run_instance |
| | block_device_mapping) as resources: |
| | File \"/usr/lib/python2.7/contextlib.py\", line 17, in __enter__ |
| | return self.gen.next() |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2261, in _build_resources |
| | reason=msg) |
| | ", "created": "2015-06-23T07:12:01Z"} |
| flavor | V1 (10) |
| hostId | b0f49b19c9d1a90b3207e2e3a88b14c36062d4a61e6e92801a3c1262 |
| id | eeca56e7-4a13-4787-9820-75f3ea07f24c |
| image | A1-SNAP1 (3da3258d-c630-4396-aca9-017f7f62d7fb) |
| key_name | - |
| metadata | {} |
| name | VIN2--eeca56e7-4a13-4787-9820-75f3ea07f24c |
| os-extended-volumes:volumes_attached | [] |
| status | ERROR |
| tenant_id | 1657ff89c9c54aa1b42d100a4a663a1f |
| updated | 2015-06-23T07:12:01Z |
| user_id | a9b90ce5977d44ec8b65ada3d31c4747 |
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
root@cs-scale-2:/var/log/nova# |
|
| 2015-06-23 22:50:04 |
OpenContrail Admin |
information type |
Proprietary |
Public |
|
| 2015-06-24 06:48:26 |
Vinay Mahuli |
juniperopenstack/r2.20: status |
New |
In Progress |
|
| 2015-06-25 15:16:30 |
Jeya ganesh babu J |
juniperopenstack/r2.20: status |
In Progress |
Fix Committed |
|
| 2015-06-29 18:45:36 |
Vinay Mahuli |
juniperopenstack/trunk: status |
New |
In Progress |
|
| 2015-11-10 00:25:38 |
Jeya ganesh babu J |
juniperopenstack/trunk: status |
In Progress |
Fix Released |
|
| 2015-11-10 00:25:43 |
Jeya ganesh babu J |
juniperopenstack/trunk: status |
Fix Released |
Fix Committed |
|
