| 2015-04-03 17:09:47 |
Miroslav Anashkin |
bug |
|
|
added bug |
| 2015-04-03 17:10:23 |
Miroslav Anashkin |
nominated for series |
|
fuel/6.1.x |
|
| 2015-04-03 17:10:23 |
Miroslav Anashkin |
bug task added |
|
fuel/6.1.x |
|
| 2015-04-03 17:10:23 |
Miroslav Anashkin |
nominated for series |
|
fuel/5.1.x |
|
| 2015-04-03 17:10:23 |
Miroslav Anashkin |
bug task added |
|
fuel/5.1.x |
|
| 2015-04-03 17:10:23 |
Miroslav Anashkin |
nominated for series |
|
fuel/6.0.x |
|
| 2015-04-03 17:10:23 |
Miroslav Anashkin |
bug task added |
|
fuel/6.0.x |
|
| 2015-04-03 17:10:30 |
Miroslav Anashkin |
fuel/6.0.x: importance |
Undecided |
High |
|
| 2015-04-03 17:10:34 |
Miroslav Anashkin |
fuel/5.1.x: importance |
Undecided |
High |
|
| 2015-04-03 17:10:55 |
Miroslav Anashkin |
fuel/6.1.x: milestone |
|
6.1 |
|
| 2015-04-03 17:11:03 |
Miroslav Anashkin |
fuel/6.0.x: milestone |
|
6.0.1 |
|
| 2015-04-03 17:11:07 |
Miroslav Anashkin |
fuel/5.1.x: milestone |
|
5.1.2 |
|
| 2015-04-03 17:11:28 |
Miroslav Anashkin |
fuel/6.1.x: assignee |
|
Fuel Library Team (fuel-library) |
|
| 2015-04-03 17:11:41 |
Miroslav Anashkin |
fuel/6.0.x: assignee |
|
Fuel Library Team (fuel-library) |
|
| 2015-04-03 17:11:54 |
Miroslav Anashkin |
fuel/5.1.x: assignee |
|
Fuel Library Team (fuel-library) |
|
| 2015-04-03 17:14:22 |
Miroslav Anashkin |
description |
We've implemented the following workaround in Fuel 5.1.1:
https://bugs.launchpad.net/fuel/+bug/1322259/
Set delay to 5.0 to recover channel errors on highly loaded environments.
Actually, this fix had been made for slow environments.
It introduced to us a new issue, affecting all Mirantis OpenStack versions between 5.1.1 and 6.1.
Instance launch may fail.
It is related to this HA+RabbitMQ+Oslo Messaging bug:
https://bugs.launchpad.net/mos/+bug/1415932
So far customers reported the following errors:
===================================================================
Error Message 1
Timed out waiting for a reply to message ID 7bc529a2d71141cf8e65cbee6402f817
Code
500
Details
File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 614, in build_instances request_spec, filter_properties) File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", line 49, in select_destinations context, request_spec, filter_properties) File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", line 35, in __run_method return getattr(self.instance, __name)(*args, **kwargs) File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/query.py", line 34, in select_destinations context, request_spec, filter_properties) File "/usr/lib/python2.7/dist-packages/nova/scheduler/rpcapi.py", line 108, in select_destinations request_spec=request_spec, filter_properties=filter_properties) File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 152, in call retry=self.retry) File "/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send timeout=timeout, retry=retry) File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 434, in send retry=retry) File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 423, in _send result = self._waiter.wait(msg_id, timeout) File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 289, in wait reply, ending = self._poll_connection(msg_id, timeout) File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 239, in _poll_connection % msg_id)
===================================================================
Error message 2
The problem I am seeing is that when cloud-init would access the metadata service it was taking 10seconds to respond to each query which is slowing down the startup times of instances from ~30sec to over 3min.
time curl http://169.254.169.254/2009-04-04/meta-data/instance-id
i-00000046
real 0m10.289s
user 0m0.000s
sys 0m0.003s
After some troubleshooting I noticed that the the neutron-metadata-agent was calling out for locks to the neutron-server, when I looked at the neutron-server logs I started seeing error messages like this:
2015-03-12 21:00:34.247 41004 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to publish message to topic 'reply_948744ac899648c19260a53a8e5c0f4b': [Errno 32] Broken pipe
===================================================================
Should we set kombu_reconnect_delay back to its default value, 1.0? |
We've implemented the following workaround in Fuel 5.1.1:
https://bugs.launchpad.net/fuel/+bug/1322259/
Set delay to 5.0 to recover channel errors on highly loaded environments.
Actually, this fix had been made for slow environments.
It introduced to us a new issue, affecting all Mirantis OpenStack versions between 5.1.1 and 6.1.
Instance launch may fail.
It is related to this HA+RabbitMQ+Oslo Messaging bug:
https://bugs.launchpad.net/mos/+bug/1415932
So far customers reported the following errors:
===================================================================
Error Message 1
Timed out waiting for a reply to message ID 7bc529a2d71141cf8e65cbee6402f817
Code
500
Details
File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 614, in build_instances request_spec, filter_properties)
File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", line 49, in select_destinations context, request_spec, filter_properties) File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", line 35, in __run_method return getattr(self.instance, __name)(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/query.py", line 34, in select_destinations context, request_spec, filter_properties)
File "/usr/lib/python2.7/dist-packages/nova/scheduler/rpcapi.py", line 108, in select_destinations request_spec=request_spec, filter_properties=filter_properties)
File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 152, in call retry=self.retry)
File "/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send timeout=timeout, retry=retry)
File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 434, in send retry=retry)
File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 423, in _send result = self._waiter.wait(msg_id, timeout)
File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 289, in wait reply, ending = self._poll_connection(msg_id, timeout)
File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 239, in _poll_connection % msg_id)
===================================================================
Error message 2
The problem I am seeing is that when cloud-init would access the metadata service it was taking 10seconds to respond to each query which is slowing down the startup times of instances from ~30sec to over 3min.
time curl http://169.254.169.254/2009-04-04/meta-data/instance-id
i-00000046
real 0m10.289s
user 0m0.000s
sys 0m0.003s
After some troubleshooting I noticed that the the neutron-metadata-agent was calling out for locks to the neutron-server, when I looked at the neutron-server logs I started seeing error messages like this:
2015-03-12 21:00:34.247 41004 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to publish message to topic 'reply_948744ac899648c19260a53a8e5c0f4b': [Errno 32] Broken pipe
===================================================================
Should we set kombu_reconnect_delay back to its default value, 1.0? |
|
| 2015-04-06 11:10:26 |
Vladimir Kuklin |
fuel/5.1.x: status |
New |
Triaged |
|
| 2015-04-06 11:10:28 |
Vladimir Kuklin |
fuel/6.0.x: status |
New |
Triaged |
|
| 2015-04-06 11:10:30 |
Vladimir Kuklin |
fuel/6.1.x: status |
New |
Triaged |
|
| 2015-04-22 10:49:27 |
Vladimir Kuklin |
tags |
customer-found |
customer-found low-hanging-fruit |
|
| 2015-04-24 11:21:13 |
Vladimir Kuklin |
fuel/6.1.x: assignee |
Fuel Library Team (fuel-library) |
Bartlomiej Piotrowski (bpiotrowski) |
|
| 2015-04-28 00:19:09 |
Mike Scherbakov |
fuel/6.1.x: assignee |
Bartlomiej Piotrowski (bpiotrowski) |
MOS Neutron (mos-neutron) |
|
| 2015-04-28 10:13:22 |
Ilya Shakhat |
fuel/6.1.x: assignee |
MOS Neutron (mos-neutron) |
|
|
| 2015-04-28 10:36:42 |
Bartłomiej Piotrowski |
marked as duplicate |
|
1410797 |
|
| 2016-04-29 21:44:27 |
Dmitry Sutyagin |
removed duplicate marker |
1410797 |
|
|
| 2016-04-29 21:48:38 |
Dmitry Sutyagin |
tags |
customer-found low-hanging-fruit |
customer-found low-hanging-fruit support |
|
| 2016-04-29 21:48:58 |
Dmitry Sutyagin |
nominated for series |
|
fuel/7.0.x |
|
| 2016-04-29 21:48:58 |
Dmitry Sutyagin |
bug task added |
|
fuel/7.0.x |
|
| 2016-04-29 21:48:58 |
Dmitry Sutyagin |
nominated for series |
|
fuel/mitaka |
|
| 2016-04-29 21:48:58 |
Dmitry Sutyagin |
bug task added |
|
fuel/mitaka |
|
| 2016-04-29 21:48:58 |
Dmitry Sutyagin |
nominated for series |
|
fuel/newton |
|
| 2016-04-29 21:48:58 |
Dmitry Sutyagin |
bug task added |
|
fuel/newton |
|
| 2016-04-29 21:48:58 |
Dmitry Sutyagin |
nominated for series |
|
fuel/future |
|
| 2016-04-29 21:48:58 |
Dmitry Sutyagin |
bug task added |
|
fuel/future |
|
| 2016-04-29 21:48:58 |
Dmitry Sutyagin |
nominated for series |
|
fuel/8.0.x |
|
| 2016-04-29 21:48:58 |
Dmitry Sutyagin |
bug task added |
|
fuel/8.0.x |
|
| 2016-04-29 21:49:08 |
Dmitry Sutyagin |
fuel/7.0.x: status |
New |
Confirmed |
|
| 2016-04-29 21:49:14 |
Dmitry Sutyagin |
fuel/7.0.x: importance |
Undecided |
High |
|
| 2016-04-29 21:49:41 |
Dmitry Sutyagin |
fuel/7.0.x: assignee |
|
MOS Neutron (mos-neutron) |
|
| 2016-04-29 21:49:47 |
Dmitry Sutyagin |
fuel/7.0.x: milestone |
|
7.0-updates |
|
| 2016-04-29 21:49:52 |
Dmitry Sutyagin |
fuel/8.0.x: milestone |
|
8.0-updates |
|
| 2016-04-29 21:50:04 |
Dmitry Sutyagin |
fuel/future: milestone |
|
next |
|
| 2016-04-29 21:50:09 |
Dmitry Sutyagin |
fuel/mitaka: milestone |
|
9.0-updates |
|
| 2016-04-29 21:50:17 |
Dmitry Sutyagin |
fuel/newton: milestone |
6.1 |
10.0 |
|
| 2016-04-29 21:51:14 |
Dmitry Sutyagin |
fuel/newton: status |
Triaged |
New |
|
| 2016-04-29 21:51:16 |
Dmitry Sutyagin |
fuel/newton: importance |
High |
Undecided |
|
| 2016-05-04 09:38:51 |
Oleksiy Molchanov |
fuel/8.0.x: status |
New |
Confirmed |
|
| 2016-05-04 09:38:54 |
Oleksiy Molchanov |
fuel/future: status |
New |
Confirmed |
|
| 2016-05-04 09:38:59 |
Oleksiy Molchanov |
fuel/8.0.x: importance |
Undecided |
High |
|
| 2016-05-04 09:39:10 |
Oleksiy Molchanov |
fuel/mitaka: importance |
Undecided |
High |
|
| 2016-05-04 09:39:13 |
Oleksiy Molchanov |
fuel/newton: importance |
Undecided |
High |
|
| 2016-05-04 09:39:18 |
Oleksiy Molchanov |
fuel/mitaka: status |
New |
Confirmed |
|
| 2016-05-04 09:39:21 |
Oleksiy Molchanov |
fuel/future: importance |
Undecided |
High |
|
| 2016-05-04 09:39:27 |
Oleksiy Molchanov |
fuel/newton: status |
New |
Confirmed |
|
| 2016-05-04 09:39:31 |
Oleksiy Molchanov |
fuel/7.0.x: status |
Confirmed |
Triaged |
|
| 2016-05-04 09:39:33 |
Oleksiy Molchanov |
fuel/8.0.x: status |
Confirmed |
Triaged |
|
| 2016-05-04 09:39:36 |
Oleksiy Molchanov |
fuel/mitaka: status |
Confirmed |
Triaged |
|
| 2016-05-04 09:39:38 |
Oleksiy Molchanov |
fuel/newton: status |
Confirmed |
Triaged |
|
| 2016-05-04 09:39:40 |
Oleksiy Molchanov |
fuel/future: status |
Confirmed |
Triaged |
|
| 2016-05-04 09:39:55 |
Oleksiy Molchanov |
fuel/newton: assignee |
|
Fuel Sustaining (fuel-sustaining-team) |
|
| 2016-05-04 09:40:07 |
Oleksiy Molchanov |
fuel/mitaka: assignee |
|
Fuel Sustaining (fuel-sustaining-team) |
|
| 2016-05-04 09:40:13 |
Oleksiy Molchanov |
fuel/future: assignee |
|
Fuel Sustaining (fuel-sustaining-team) |
|
| 2016-05-04 09:40:19 |
Oleksiy Molchanov |
fuel/8.0.x: assignee |
|
Fuel Sustaining (fuel-sustaining-team) |
|
| 2016-05-04 09:40:24 |
Oleksiy Molchanov |
fuel/7.0.x: assignee |
MOS Neutron (mos-neutron) |
Fuel Sustaining (fuel-sustaining-team) |
|
| 2016-05-04 09:40:30 |
Oleksiy Molchanov |
fuel/6.1.x: assignee |
|
Fuel Sustaining (fuel-sustaining-team) |
|
| 2016-05-04 09:40:36 |
Oleksiy Molchanov |
fuel/6.0.x: assignee |
Fuel Library (Deprecated) (fuel-library) |
Fuel Sustaining (fuel-sustaining-team) |
|
| 2016-05-04 09:40:41 |
Oleksiy Molchanov |
fuel/5.1.x: assignee |
Fuel Library (Deprecated) (fuel-library) |
Fuel Sustaining (fuel-sustaining-team) |
|
| 2016-05-06 17:13:55 |
Dmitry Pyzhov |
tags |
customer-found low-hanging-fruit support |
area-library customer-found low-hanging-fruit support |
|
| 2016-05-13 10:15:20 |
Dmitry Pyzhov |
bug task deleted |
fuel/future |
|
|
| 2016-05-13 10:15:38 |
Dmitry Pyzhov |
bug task deleted |
fuel/newton |
|
|
| 2016-05-13 10:15:59 |
Dmitry Pyzhov |
fuel/mitaka: milestone |
9.0-updates |
9.0 |
|
| 2016-05-13 10:16:07 |
Dmitry Pyzhov |
fuel/mitaka: assignee |
Fuel Sustaining (fuel-sustaining-team) |
Networking (l23-network) |
|
| 2016-05-13 10:16:20 |
Dmitry Pyzhov |
fuel: assignee |
Fuel Sustaining (fuel-sustaining-team) |
Networking (l23-network) |
|
| 2016-05-13 10:21:35 |
Ivan Berezovskiy |
fuel/mitaka: status |
Triaged |
Fix Committed |
|
| 2016-05-13 10:21:39 |
Ivan Berezovskiy |
fuel: status |
Triaged |
Fix Committed |
|
| 2016-05-13 10:21:59 |
Ivan Berezovskiy |
fuel/8.0.x: assignee |
Fuel Sustaining (fuel-sustaining-team) |
Alexey Deryugin (velovec) |
|
| 2016-05-13 10:22:04 |
Ivan Berezovskiy |
fuel/7.0.x: assignee |
Fuel Sustaining (fuel-sustaining-team) |
Alexey Deryugin (velovec) |
|
| 2016-05-13 10:22:10 |
Ivan Berezovskiy |
fuel/6.1.x: assignee |
Fuel Sustaining (fuel-sustaining-team) |
Alexey Deryugin (velovec) |
|
| 2016-05-13 10:22:16 |
Ivan Berezovskiy |
fuel/6.0.x: assignee |
Fuel Sustaining (fuel-sustaining-team) |
Alexey Deryugin (velovec) |
|
| 2016-05-13 10:22:21 |
Ivan Berezovskiy |
fuel/5.1.x: assignee |
Fuel Sustaining (fuel-sustaining-team) |
Alexey Deryugin (velovec) |
|
| 2016-05-13 10:22:36 |
Ivan Berezovskiy |
fuel/mitaka: assignee |
Networking (l23-network) |
Ivan Berezovskiy (iberezovskiy) |
|
| 2016-05-13 10:22:38 |
Ivan Berezovskiy |
fuel: assignee |
Networking (l23-network) |
Ivan Berezovskiy (iberezovskiy) |
|
| 2016-05-25 08:42:37 |
Vitaly Sedelnik |
fuel/8.0.x: milestone |
8.0-updates |
8.0-mu-2 |
|
| 2016-06-15 09:27:52 |
Kyrylo Romanenko |
tags |
area-library customer-found low-hanging-fruit support |
area-library customer-found low-hanging-fruit on-verification support |
|
| 2016-06-16 10:28:47 |
Vitaly Sedelnik |
fuel/8.0.x: status |
Triaged |
Fix Committed |
|
| 2016-06-17 06:20:00 |
Alexey Deryugin |
fuel/5.1.x: status |
Triaged |
In Progress |
|
| 2016-06-17 06:20:03 |
Alexey Deryugin |
fuel/6.0.x: status |
Triaged |
In Progress |
|
| 2016-06-17 06:20:06 |
Alexey Deryugin |
fuel/7.0.x: status |
Triaged |
In Progress |
|
| 2016-06-17 06:20:10 |
Alexey Deryugin |
fuel/6.1.x: status |
Triaged |
In Progress |
|
| 2016-06-17 10:42:42 |
Kyrylo Romanenko |
fuel/mitaka: status |
Fix Committed |
Fix Released |
|
| 2016-06-17 10:42:55 |
Kyrylo Romanenko |
tags |
area-library customer-found low-hanging-fruit on-verification support |
area-library customer-found low-hanging-fruit support |
|
| 2016-06-27 10:53:54 |
Ekaterina Shutova |
tags |
area-library customer-found low-hanging-fruit support |
area-library customer-found low-hanging-fruit on-verification support |
|
| 2016-06-27 15:05:10 |
Ekaterina Shutova |
tags |
area-library customer-found low-hanging-fruit on-verification support |
area-library customer-found low-hanging-fruit support |
|
| 2016-06-27 15:05:16 |
Ekaterina Shutova |
fuel/8.0.x: status |
Fix Committed |
Fix Released |
|
| 2016-07-19 14:27:33 |
Alexey Deryugin |
fuel/7.0.x: status |
In Progress |
Fix Committed |
|
| 2016-07-19 14:27:36 |
Alexey Deryugin |
fuel/6.1.x: status |
In Progress |
Fix Committed |
|
| 2016-10-21 12:07:06 |
Alexey Deryugin |
fuel/5.1.x: assignee |
Alexey Deryugin (velovec) |
|
|
| 2016-10-21 12:07:09 |
Alexey Deryugin |
fuel/6.0.x: assignee |
Alexey Deryugin (velovec) |
|
|
| 2016-10-21 12:07:14 |
Alexey Deryugin |
fuel/5.1.x: status |
In Progress |
Confirmed |
|
| 2016-10-21 12:07:23 |
Alexey Deryugin |
fuel/6.0.x: status |
In Progress |
Confirmed |
|
| 2016-10-25 13:10:21 |
Oleksiy Molchanov |
fuel/5.1.x: assignee |
|
MOS Maintenance (mos-maintenance) |
|
| 2016-10-25 13:10:25 |
Oleksiy Molchanov |
fuel/6.0.x: assignee |
|
MOS Maintenance (mos-maintenance) |
|
| 2017-04-10 09:07:10 |
Ekaterina Shutova |
fuel: status |
Fix Committed |
Fix Released |
|
| 2017-05-29 09:00:26 |
Alexey Stupnikov |
fuel/5.1.x: status |
Confirmed |
Won't Fix |
|
| 2017-05-29 09:00:36 |
Alexey Stupnikov |
fuel/6.0.x: status |
Confirmed |
Won't Fix |
|
