[All] Add oslo rabbitmq heartbeat settings

Bug #1462438 reported by Mark Vanderwiel
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack + Chef
Fix Released
Medium
Ma Wen Cheng

Bug Description

When using rabbit mq in multi controller environment (HA), if a controller node (with rabbitmq service) goes down, it can take a a long time (15 minutes) for compute nodes to automatically recover and begin using the alternate rabbitmq service. There existing heartbeat configuration options in oslo rabbitmq support to better handle these situations.

https://github.com/openstack/oslo.messaging/blob/d685e6f80a5dfc5fba638beacde762c8ccf9a89d/oslo_messaging/_drivers/impl_rabbit.py#L138

cfg.IntOpt('heartbeat_timeout_threshold',
default=0,
help="Number of seconds after which the Rabbit broker is "
"considered down if heartbeat's keep-alive fails "
"(0 disable the heartbeat). EXPERIMENTAL"),

cfg.IntOpt('heartbeat_rate',
default=2,
help='How often times during the heartbeat_timeout_threshold '
'we check the heartbeat.'),

These will be added to Common messaging attributes, and then to all the cookbook that uses rabbitmq.

cookbook-openstack-bare-metal/templates/default/ironic.conf.erb:[oslo_messaging_rabbit]
cookbook-openstack-block-storage/templates/default/cinder.conf.erb:[oslo_messaging_rabbit]
cookbook-openstack-compute/templates/default/nova.conf.erb:[oslo_messaging_rabbit]
cookbook-openstack-identity/templates/default/keystone.conf.erb:[oslo_messaging_rabbit]
cookbook-openstack-image/templates/default/glance-api.conf.erb:[oslo_messaging_rabbit]
cookbook-openstack-image/templates/default/glance-registry.conf.erb:[oslo_messaging_rabbit]
cookbook-openstack-network/templates/default/neutron.conf.erb:[oslo_messaging_rabbit]
cookbook-openstack-orchestration/templates/default/heat.conf.erb:[oslo_messaging_rabbit]

Tags: all
description: updated
Changed in openstack-chef:
importance: Undecided → Medium
milestone: none → kilo-rc1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cookbook-openstack-common (master)

Fix proposed to branch: master
Review: https://review.openstack.org/188850

Changed in openstack-chef:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cookbook-openstack-bare-metal (master)

Fix proposed to branch: master
Review: https://review.openstack.org/188851

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cookbook-openstack-block-storage (master)

Fix proposed to branch: master
Review: https://review.openstack.org/188852

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cookbook-openstack-compute (master)

Fix proposed to branch: master
Review: https://review.openstack.org/188855

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cookbook-openstack-identity (master)

Fix proposed to branch: master
Review: https://review.openstack.org/188858

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cookbook-openstack-image (master)

Fix proposed to branch: master
Review: https://review.openstack.org/188861

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cookbook-openstack-network (master)

Fix proposed to branch: master
Review: https://review.openstack.org/188865

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cookbook-openstack-orchestration (master)

Fix proposed to branch: master
Review: https://review.openstack.org/188867

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cookbook-openstack-telemetry (master)

Fix proposed to branch: master
Review: https://review.openstack.org/188894

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cookbook-openstack-common (master)

Reviewed: https://review.openstack.org/188850
Committed: https://git.openstack.org/cgit/stackforge/cookbook-openstack-common/commit/?id=fafc3c83ac24c954c16828dcc2a7fba1b25924a0
Submitter: Jenkins
Branch: master

commit fafc3c83ac24c954c16828dcc2a7fba1b25924a0
Author: Mark Vanderwiel <email address hidden>
Date: Fri Jun 5 11:37:04 2015 -0500

    Allow oslo rabbit heartbeat configuration

    In order to allow components to better handle and respond to mq failures,
    oslo has some heartbeat options that are useful.

    Change-Id: I75b2b5e140095bdc828df6af13159170aa08839c
    Partial-Bug: #1462438

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cookbook-openstack-compute (master)

Reviewed: https://review.openstack.org/188855
Committed: https://git.openstack.org/cgit/stackforge/cookbook-openstack-compute/commit/?id=51886f084aafce6b4720d3539ec9c3296403e1db
Submitter: Jenkins
Branch: master

commit 51886f084aafce6b4720d3539ec9c3296403e1db
Author: Mark Vanderwiel <email address hidden>
Date: Fri Jun 5 11:58:15 2015 -0500

    Allow oslo rabbit heartbeat configuration

    In order to allow components to better handle and respond to mq failures,
    oslo has some heartbeat options that are useful.

    Change-Id: Ib69e69e1ace08448d789f3eb28749d0d228ae00d
    Partial-Bug: #1462438

Changed in openstack-chef:
assignee: Mark Vanderwiel (vanderwl) → Ma Wen Cheng (mars914)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cookbook-openstack-orchestration (master)

Reviewed: https://review.openstack.org/188867
Committed: https://git.openstack.org/cgit/openstack/cookbook-openstack-orchestration/commit/?id=a84a1443391db5da3161790d92bb8256fd7f5d6b
Submitter: Jenkins
Branch: master

commit a84a1443391db5da3161790d92bb8256fd7f5d6b
Author: Mark Vanderwiel <email address hidden>
Date: Fri Jun 5 12:13:57 2015 -0500

    Allow oslo rabbit heartbeat configuration

    In order to allow components to better handle and respond to mq failures,
    oslo has some heartbeat options that are useful.

    Change-Id: I453b04c90e6caba7c00cf239035af05199bb5c36
    Partial-Bug: #1462438

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cookbook-openstack-identity (master)

Reviewed: https://review.openstack.org/188858
Committed: https://git.openstack.org/cgit/openstack/cookbook-openstack-identity/commit/?id=ccac1491cf8c806dc5b17717c7cab36413189b55
Submitter: Jenkins
Branch: master

commit ccac1491cf8c806dc5b17717c7cab36413189b55
Author: Mark Vanderwiel <email address hidden>
Date: Fri Jun 5 12:01:19 2015 -0500

    Allow oslo rabbit heartbeat configuration

    In order to allow components to better handle and respond to mq failures,
    oslo has some heartbeat options that are useful.

    Change-Id: Ied6ad4af67f8915e91c7515f43121da9ee294456
    Partial-Bug: #1462438

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cookbook-openstack-telemetry (master)

Reviewed: https://review.openstack.org/188894
Committed: https://git.openstack.org/cgit/openstack/cookbook-openstack-telemetry/commit/?id=430f5f746633c7dc0a59e43f32fac2e317881d31
Submitter: Jenkins
Branch: master

commit 430f5f746633c7dc0a59e43f32fac2e317881d31
Author: Mark Vanderwiel <email address hidden>
Date: Fri Jun 5 14:16:51 2015 -0500

    Allow oslo rabbit heartbeat configuration

    In order to allow components to better handle and respond to mq failures,
    oslo has some heartbeat options that are useful.

    Change-Id: I02faeba2c10d9c9b8035819156bfdc0759bafcc6
    Partial-Bug: #1462438

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cookbook-openstack-image (master)

Reviewed: https://review.openstack.org/188861
Committed: https://git.openstack.org/cgit/openstack/cookbook-openstack-image/commit/?id=2f6516755acfa346c49e19f9e95a04afa82e47be
Submitter: Jenkins
Branch: master

commit 2f6516755acfa346c49e19f9e95a04afa82e47be
Author: Mark Vanderwiel <email address hidden>
Date: Fri Jun 5 12:05:10 2015 -0500

    Allow oslo rabbit heartbeat configuration

    In order to allow components to better handle and respond to mq failures,
    oslo has some heartbeat options that are useful.

    Change-Id: I41b3fc4699a4f285e9fcb066da55bb8a7f1361a2
    Partial-Bug: #1462438

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cookbook-openstack-block-storage (master)

Reviewed: https://review.openstack.org/188852
Committed: https://git.openstack.org/cgit/openstack/cookbook-openstack-block-storage/commit/?id=9cc96cd64e752f9df41cab21b727c7ab3a860877
Submitter: Jenkins
Branch: master

commit 9cc96cd64e752f9df41cab21b727c7ab3a860877
Author: Mark Vanderwiel <email address hidden>
Date: Fri Jun 5 11:55:26 2015 -0500

    Allow oslo rabbit heartbeat configuration

    In order to allow components to better handle and respond to mq failures,
    oslo has some heartbeat options that are useful.

    Change-Id: If21b39cb0c640fea0846f448100842374f97f550
    Partial-Bug: #1462438

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cookbook-openstack-network (master)

Reviewed: https://review.openstack.org/188865
Committed: https://git.openstack.org/cgit/openstack/cookbook-openstack-network/commit/?id=4b0d6b1ec0960d6ffb635adff324737dadc137e5
Submitter: Jenkins
Branch: master

commit 4b0d6b1ec0960d6ffb635adff324737dadc137e5
Author: Mark Vanderwiel <email address hidden>
Date: Fri Jun 5 12:08:19 2015 -0500

    Allow oslo rabbit heartbeat configuration

    In order to allow components to better handle and respond to mq failures,
    oslo has some heartbeat options that are useful.

    Change-Id: Ief96c7fce659376f476b11b527c60ce999777b81
    Partial-Bug: #1462438

Changed in openstack-chef:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.