[all] Can not set rabbit_max_retries and rabbit_retry_interval

Bug #1439968 reported by Wei Hu on 2015-04-03
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack + Chef
Medium
Ma Wen Cheng

Bug Description

When using rabbitmq as the AMQP service, user can not assign rabbit_max_retries and rabbit_retry_interval value. rabbit_max_reties default to 0 which will let oslo.message retry forever when disconnect with rabbitmq-server. Sometimes, this is very dangerous, because those retry and AMQP queue can not connect error log will fill the whole disk.

Tags: all Edit Tag help
Wei Hu (huwei-xtu) on 2015-04-03
Changed in openstack-chef:
assignee: nobody → Wei Hu (huwei-xtu)
Wei Hu (huwei-xtu) wrote :

See the options usage in oslo.message code.
https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L106

But I can find those options only in neutron.conf. The glance and nova relate configuration files do not contain those options.
https://github.com/openstack/glance/blob/master/etc/glance-registry.conf#L98

I guess those options should also be included in those configuration files. But not sure.

Fix proposed to branch: master
Review: https://review.openstack.org/171074

Changed in openstack-chef:
status: New → In Progress

Reviewed: https://review.openstack.org/171074
Committed: https://git.openstack.org/cgit/stackforge/cookbook-openstack-common/commit/?id=22c46294d6d5623f7a0b2c162c0aef69cb3be760
Submitter: Jenkins
Branch: master

commit 22c46294d6d5623f7a0b2c162c0aef69cb3be760
Author: Wei Hu <email address hidden>
Date: Tue Apr 7 15:41:55 2015 +0800

    Add rabbitmq max_retries and retry_interval attr

    Oslo.message using rabbit_max_retries and rabbit_retry_interval to
    define reconnect rabbitmq server times and interval when can not
    connect to rabbitmq server.

    Change-Id: I6bba768f4cf0c61b92fbc5d682b60f094afb7180
    Partial-bug: #1439968

Changed in openstack-chef:
assignee: Wei Hu (huwei-xtu) → Ma Wen Cheng (mars914)

Reviewed: https://review.openstack.org/171087
Committed: https://git.openstack.org/cgit/stackforge/cookbook-openstack-network/commit/?id=0becd087221b63ace01f4d80a465fc303885e64a
Submitter: Jenkins
Branch: master

commit 0becd087221b63ace01f4d80a465fc303885e64a
Author: Wei Hu <email address hidden>
Date: Tue Apr 7 16:50:42 2015 +0800

    Add rabbitmq max_retries and retry_interval in neutron.conf.erb

    Oslo.message using rabbit_max_retries and rabbit_retry_interval to
    define reconnect rabbitmq server times and interval when can not
    connect to rabbitmq server.

    Partial-bug: #1439968

    Change-Id: I8e19c3343627d4e2316f91b4fb084c5b6486dd84

Ma Wen Cheng (mars914) on 2015-04-20
Changed in openstack-chef:
assignee: Ma Wen Cheng (mars914) → nobody

it is also needed to update all the other components for these two attributes.

tags: added: all
Wei Hu (huwei-xtu) on 2015-05-04
Changed in openstack-chef:
assignee: nobody → Wei Hu (huwei-xtu)
Changed in openstack-chef:
importance: Undecided → Medium
milestone: none → kilo-rc1
summary: - Can not set rabbit_max_retries and rabbit_retry_interval
+ [all] Can not set rabbit_max_retries and rabbit_retry_interval

Reviewed: https://review.openstack.org/182173
Committed: https://git.openstack.org/cgit/stackforge/cookbook-openstack-compute/commit/?id=8d1f4228d27c1ebfbebb20f3c6f9ccca6efb60e7
Submitter: Jenkins
Branch: master

commit 8d1f4228d27c1ebfbebb20f3c6f9ccca6efb60e7
Author: Wei Hu <email address hidden>
Date: Tue May 12 15:05:50 2015 +0800

    Add rabbitmq max_retries and retry_interval in nova.conf.erb

    Oslo.message using rabbit_max_retries and rabbit_retry_interval to
    define reconnect rabbitmq server times and interval when can not
    connect to rabbitmq server.

    Change-Id: Ie5ce28de16ed2b4b9c51a8b81e4accaca1466cac
    Partial-bug: #1439968

Wei Hu (huwei-xtu) wrote :

Fix nova and neutron. Keystone does not need this option.

Still have glance, ceilometer. cinder components need fix.

Fix proposed to branch: master
Review: https://review.openstack.org/190948

Changed in openstack-chef:
assignee: Wei Hu (huwei-xtu) → Ma Wen Cheng (mars914)
Ma Wen Cheng (mars914) wrote :

Based on the current codes, keystone also needs these two options.

Ma Wen Cheng (mars914) wrote :

I will fix the left components instead of Wei Hu to move on this bug.

Reviewed: https://review.openstack.org/190948
Committed: https://git.openstack.org/cgit/stackforge/cookbook-openstack-identity/commit/?id=141b2738b443873b0295870f75c9a2d16ab9c0d0
Submitter: Jenkins
Branch: master

commit 141b2738b443873b0295870f75c9a2d16ab9c0d0
Author: wenchma <email address hidden>
Date: Fri Jun 12 14:21:50 2015 +0800

    Add rabbitmq max_retries and retry_interval attributes in keystone conf

    Oslo.message using rabbit_max_retries and rabbit_retry_interval to
    define reconnect rabbitmq server times and interval when can not
    connect to rabbitmq server.

    Change-Id: I54236d1d0243e73eb27dd8984cfb86f8bbf7a271
    Closes-bug: #1439968

Changed in openstack-chef:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/190957
Committed: https://git.openstack.org/cgit/stackforge/cookbook-openstack-orchestration/commit/?id=5259486ff4fde915e0f3f866b6d9c44fc98dedeb
Submitter: Jenkins
Branch: master

commit 5259486ff4fde915e0f3f866b6d9c44fc98dedeb
Author: wenchma <email address hidden>
Date: Fri Jun 12 15:01:46 2015 +0800

    Add rabbitmq max_retries and retry_interval in heat conf

    Oslo.message using rabbit_max_retries and rabbit_retry_interval to
    define reconnect rabbitmq server times and interval when can not
    connect to rabbitmq server.

    Change-Id: I261d3f31aa89f845d88ee4cbd1acb56c9131ba47
    Closes-bug: #1439968

Reviewed: https://review.openstack.org/190961
Committed: https://git.openstack.org/cgit/stackforge/cookbook-openstack-telemetry/commit/?id=ff2d6f597dfc46a87b95dc6eeb8cd7c9fdcc7e2c
Submitter: Jenkins
Branch: master

commit ff2d6f597dfc46a87b95dc6eeb8cd7c9fdcc7e2c
Author: wenchma <email address hidden>
Date: Fri Jun 12 15:14:47 2015 +0800

    Add rabbitmq max_retries and retry_interval attributes in ceilometer conf

    Oslo.message using rabbit_max_retries and rabbit_retry_interval to
    define reconnect rabbitmq server times and interval when can not
    connect to rabbitmq server.

    Change-Id: I1fd9d4840a55453667eb2ed4c03e4c3cb72442b9
    Closes-bug: #1439968

Reviewed: https://review.openstack.org/190951
Committed: https://git.openstack.org/cgit/stackforge/cookbook-openstack-image/commit/?id=c46945c8e98244f8c94b5d08dfdfbfb95f5b6c3d
Submitter: Jenkins
Branch: master

commit c46945c8e98244f8c94b5d08dfdfbfb95f5b6c3d
Author: wenchma <email address hidden>
Date: Fri Jun 12 14:35:41 2015 +0800

    Add rabbitmq max_retries and retry_interval attributes in glance conf

    Oslo.message using rabbit_max_retries and rabbit_retry_interval to
    define reconnect rabbitmq server times and interval when can not
    connect to rabbitmq server.

    Change-Id: Id205952829678a83b0ccd8e415c64cf1283e0e08
    Closes-bug: #1439968

Reviewed: https://review.openstack.org/190955
Committed: https://git.openstack.org/cgit/stackforge/cookbook-openstack-block-storage/commit/?id=3c0ceb9f4cd47b6030e9399f380a998bb0f89fc7
Submitter: Jenkins
Branch: master

commit 3c0ceb9f4cd47b6030e9399f380a998bb0f89fc7
Author: wenchma <email address hidden>
Date: Fri Jun 12 14:48:03 2015 +0800

    Add rabbitmq max_retries and retry_interval in cinder conf

    Oslo.message using rabbit_max_retries and rabbit_retry_interval to
    define reconnect rabbitmq server times and interval when can not
    connect to rabbitmq server.

    Change-Id: I449f7be68dd9c9adbad587a47cf9d9dcc3462369
    Closes-bug: #1439968

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers