| 2017-06-09 04:21:03 |
Sanchit Malhotra |
bug |
|
|
added bug |
| 2017-06-09 06:51:25 |
Sanchit Malhotra |
description |
When Ceilometer Middleware is integrated with SWIFT, some of user requests hangs at ceilometer level.
When we debugged firther, to identify where the requests are stuck, it is observed that the following call of oslo_messaging Notifier class does not return.
{code}
47 import oslo_messaging
....
121 self._notifier = oslo_messaging.Notifier(
122 oslo_messaging.get_transport(cfg.CONF, url=conf.get('url')),
123 publisher_id='ceilometermiddleware',
124 driver=conf.get('driver', 'messagingv2'),
125 topic=conf.get('topic', 'notifications'))
126
....
283
284 self._notifier.info(context.get_admin_context().to_dict(),
285 'objectstore.http.request', event.as_dict())
286
{code}
the call "self._notifier.info" does not return in some case and due to this, the user requests are stuck indefinitely.
It is confirmed that there is no problem with RabbitMQ server or ceilometer server as both of them are accessible from other SWIFT cluster at the time problem is observed on different setup. |
When Ceilometer Middleware is integrated with SWIFT, some of user requests hangs at ceilometer level.
When we debugged firther, to identify where the requests are stuck, it is observed that the following call of oslo_messaging Notifier class does not return.
{code}
47 import oslo_messaging
....
121 self._notifier = oslo_messaging.Notifier(
122 oslo_messaging.get_transport(cfg.CONF, url=conf.get('url')),
123 publisher_id='ceilometermiddleware',
124 driver=conf.get('driver', 'messagingv2'),
125 topic=conf.get('topic', 'notifications'))
126
....
283
284 self._notifier.info(context.get_admin_context().to_dict(),
285 'objectstore.http.request', event.as_dict())
286
{code}
the call "self._notifier.info" does not return in some case and due to this, the user requests are stuck indefinitely.
It is confirmed that there is no problem with RabbitMQ server or ceilometer server as both of them are accessible from other SWIFT cluster at the time problem is observed on different setup.
When conformed from network packet dump, it is seen that ceilometer middleware does not sends any data(metering information) to rabbitMQ server.
It is thus confirmed that the request is stuck somewhere in oslo_messaging notifier API call.
Also, it's effect is seen in Proxy Service as no new user requests are handled once any of the request is stuck in such a manner. |
|
| 2017-06-09 06:51:56 |
Sanchit Malhotra |
bug task added |
|
ceilometermiddleware |
|
| 2017-06-09 06:52:32 |
Sanchit Malhotra |
description |
When Ceilometer Middleware is integrated with SWIFT, some of user requests hangs at ceilometer level.
When we debugged firther, to identify where the requests are stuck, it is observed that the following call of oslo_messaging Notifier class does not return.
{code}
47 import oslo_messaging
....
121 self._notifier = oslo_messaging.Notifier(
122 oslo_messaging.get_transport(cfg.CONF, url=conf.get('url')),
123 publisher_id='ceilometermiddleware',
124 driver=conf.get('driver', 'messagingv2'),
125 topic=conf.get('topic', 'notifications'))
126
....
283
284 self._notifier.info(context.get_admin_context().to_dict(),
285 'objectstore.http.request', event.as_dict())
286
{code}
the call "self._notifier.info" does not return in some case and due to this, the user requests are stuck indefinitely.
It is confirmed that there is no problem with RabbitMQ server or ceilometer server as both of them are accessible from other SWIFT cluster at the time problem is observed on different setup.
When conformed from network packet dump, it is seen that ceilometer middleware does not sends any data(metering information) to rabbitMQ server.
It is thus confirmed that the request is stuck somewhere in oslo_messaging notifier API call.
Also, it's effect is seen in Proxy Service as no new user requests are handled once any of the request is stuck in such a manner. |
When Ceilometer Middleware is integrated with SWIFT, some of user requests hangs at ceilometer level.
When we debugged firther, to identify where the requests are stuck, it is observed that the following call of oslo_messaging Notifier class does not return.
File: https://github.com/openstack/ceilometermiddleware/blob/stable/mitaka/ceilometermiddleware/swift.py#L248
{code}
47 import oslo_messaging
....
121 self._notifier = oslo_messaging.Notifier(
122 oslo_messaging.get_transport(cfg.CONF, url=conf.get('url')),
123 publisher_id='ceilometermiddleware',
124 driver=conf.get('driver', 'messagingv2'),
125 topic=conf.get('topic', 'notifications'))
126
....
283
284 self._notifier.info(context.get_admin_context().to_dict(),
285 'objectstore.http.request', event.as_dict())
286
{code}
the call "self._notifier.info" does not return in some case and due to this, the user requests are stuck indefinitely.
It is confirmed that there is no problem with RabbitMQ server or ceilometer server as both of them are accessible from other SWIFT cluster at the time problem is observed on different setup.
When conformed from network packet dump, it is seen that ceilometer middleware does not sends any data(metering information) to rabbitMQ server.
It is thus confirmed that the request is stuck somewhere in oslo_messaging notifier API call.
Also, it's effect is seen in Proxy Service as no new user requests are handled once any of the request is stuck in such a manner. |
|
| 2017-06-12 11:16:46 |
Sanchit Malhotra |
description |
When Ceilometer Middleware is integrated with SWIFT, some of user requests hangs at ceilometer level.
When we debugged firther, to identify where the requests are stuck, it is observed that the following call of oslo_messaging Notifier class does not return.
File: https://github.com/openstack/ceilometermiddleware/blob/stable/mitaka/ceilometermiddleware/swift.py#L248
{code}
47 import oslo_messaging
....
121 self._notifier = oslo_messaging.Notifier(
122 oslo_messaging.get_transport(cfg.CONF, url=conf.get('url')),
123 publisher_id='ceilometermiddleware',
124 driver=conf.get('driver', 'messagingv2'),
125 topic=conf.get('topic', 'notifications'))
126
....
283
284 self._notifier.info(context.get_admin_context().to_dict(),
285 'objectstore.http.request', event.as_dict())
286
{code}
the call "self._notifier.info" does not return in some case and due to this, the user requests are stuck indefinitely.
It is confirmed that there is no problem with RabbitMQ server or ceilometer server as both of them are accessible from other SWIFT cluster at the time problem is observed on different setup.
When conformed from network packet dump, it is seen that ceilometer middleware does not sends any data(metering information) to rabbitMQ server.
It is thus confirmed that the request is stuck somewhere in oslo_messaging notifier API call.
Also, it's effect is seen in Proxy Service as no new user requests are handled once any of the request is stuck in such a manner. |
When Ceilometer Middleware is integrated with SWIFT, some of user requests hangs at ceilometer level.
When we debugged firther, to identify where the requests are stuck, it is observed that the following call of oslo_messaging Notifier class does not return.
File: https://github.com/openstack/ceilometermiddleware/blob/stable/mitaka/ceilometermiddleware/swift.py#L248
{code}
47 import oslo_messaging
....
121 self._notifier = oslo_messaging.Notifier(
122 oslo_messaging.get_transport(cfg.CONF, url=conf.get('url')),
123 publisher_id='ceilometermiddleware',
124 driver=conf.get('driver', 'messagingv2'),
125 topic=conf.get('topic', 'notifications'))
126
....
283
284 self._notifier.info(context.get_admin_context().to_dict(),
285 'objectstore.http.request', event.as_dict())
286
{code}
the call "self._notifier.info" does not return in some case and due to this, the user requests are stuck indefinitely.
It is confirmed that there is no problem with RabbitMQ server or ceilometer server as both of them are accessible from other SWIFT cluster at the time problem is observed on different setup.
When conformed from network packet dump, it is seen that ceilometer middleware does not sends any data(metering information) to rabbitMQ server.
It is thus confirmed that the request is stuck somewhere in oslo_messaging notifier API call.
Also, it's effect is seen in Proxy Service as no new user requests are handled once any of the request is stuck in such a manner.
On restart of Proxy Service, the problem seems to be resolved temporarily. |
|
| 2017-06-14 04:11:53 |
Sanchit Malhotra |
bug watch added |
|
http://bugs.python.org/issue8844 |
|
| 2017-07-07 03:20:21 |
Sanchit Malhotra |
description |
When Ceilometer Middleware is integrated with SWIFT, some of user requests hangs at ceilometer level.
When we debugged firther, to identify where the requests are stuck, it is observed that the following call of oslo_messaging Notifier class does not return.
File: https://github.com/openstack/ceilometermiddleware/blob/stable/mitaka/ceilometermiddleware/swift.py#L248
{code}
47 import oslo_messaging
....
121 self._notifier = oslo_messaging.Notifier(
122 oslo_messaging.get_transport(cfg.CONF, url=conf.get('url')),
123 publisher_id='ceilometermiddleware',
124 driver=conf.get('driver', 'messagingv2'),
125 topic=conf.get('topic', 'notifications'))
126
....
283
284 self._notifier.info(context.get_admin_context().to_dict(),
285 'objectstore.http.request', event.as_dict())
286
{code}
the call "self._notifier.info" does not return in some case and due to this, the user requests are stuck indefinitely.
It is confirmed that there is no problem with RabbitMQ server or ceilometer server as both of them are accessible from other SWIFT cluster at the time problem is observed on different setup.
When conformed from network packet dump, it is seen that ceilometer middleware does not sends any data(metering information) to rabbitMQ server.
It is thus confirmed that the request is stuck somewhere in oslo_messaging notifier API call.
Also, it's effect is seen in Proxy Service as no new user requests are handled once any of the request is stuck in such a manner.
On restart of Proxy Service, the problem seems to be resolved temporarily. |
When Ceilometer Middleware is integrated with SWIFT, some of user requests hangs at ceilometer level.
It is observed that with default max pool size of 30, in case if 31 requests are received at oslo_messaging, the hung behaviour (indefinite hung) is observed.
When we debugged firther, to identify where the requests are stuck, it is observed that the following call of oslo_messaging Notifier class does not return.
File: https://github.com/openstack/ceilometermiddleware/blob/stable/mitaka/ceilometermiddleware/swift.py#L248
{code}
47 import oslo_messaging
....
121 self._notifier = oslo_messaging.Notifier(
122 oslo_messaging.get_transport(cfg.CONF, url=conf.get('url')),
123 publisher_id='ceilometermiddleware',
124 driver=conf.get('driver', 'messagingv2'),
125 topic=conf.get('topic', 'notifications'))
126
....
283
284 self._notifier.info(context.get_admin_context().to_dict(),
285 'objectstore.http.request', event.as_dict())
286
{code}
the call "self._notifier.info" does not return in some case and due to this, the user requests are stuck indefinitely.
It is confirmed that there is no problem with RabbitMQ server or ceilometer server as both of them are accessible from other SWIFT cluster at the time problem is observed on different setup.
When conformed from network packet dump, it is seen that ceilometer middleware does not sends any data(metering information) to rabbitMQ server.
It is thus confirmed that the request is stuck somewhere in oslo_messaging notifier API call.
Also, it's effect is seen in Proxy Service as no new user requests are handled once any of the request is stuck in such a manner.
On restart of Proxy Service, the problem seems to be resolved temporarily. |
|
| 2017-10-10 18:02:50 |
Ken Giusti |
bug watch added |
|
https://github.com/eventlet/eventlet/issues/428 |
|
| 2018-04-25 22:15:51 |
Ben Nemec |
oslo.messaging: status |
New |
Incomplete |
|
