Default for qpid_heartbeat should be increased

Reported by Russell Bryant on 2012-09-13
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
High
Russell Bryant
Glance
Medium
Russell Bryant
OpenStack Compute (nova)
High
Russell Bryant
neutron
High
Russell Bryant
openstack-common
Fix Released
High
Russell Bryant

Bug Description

The heartbeat option enables sending a message at the AMQP layer at a
regular interval. This helps ensure that the connection is still up and
working. Once a failure is detected, a reconnection can be attempted.

The default for qpid_heartbeat was previously 5 seconds. This is pretty
aggressive and can lead to invalid timeouts occuring. An example is the
nova-compute service. It may block in native code (such as libvirt)
under normal circumstances. When this happens, other greenthreads (such
as the one responsible for sending the heartbeat) can not be scheduled.
It should be increased to something more reasonable, like 1 minute.

Changed in nova:
assignee: nobody → Russell Bryant (russellb)
milestone: none → folsom-rc1
importance: Undecided → High
status: New → In Progress
status: In Progress → Confirmed
Changed in openstack-common:
status: New → In Progress
importance: Undecided → High
assignee: nobody → Russell Bryant (russellb)
Changed in cinder:
milestone: none → folsom-rc1
assignee: nobody → Russell Bryant (russellb)
importance: Undecided → High
status: New → Confirmed
Changed in quantum:
status: New → Confirmed
assignee: nobody → Russell Bryant (russellb)

Reviewed: https://review.openstack.org/12981
Committed: http://github.com/openstack/openstack-common/commit/1a9d5c8df5aeee4faf99b54800566e386166836d
Submitter: Jenkins
Branch: master

commit 1a9d5c8df5aeee4faf99b54800566e386166836d
Author: Russell Bryant <email address hidden>
Date: Thu Sep 13 16:12:41 2012 -0400

    Change the qpid_heartbeat default to 60 seconds.

    The heartbeat option enables sending a message at the AMQP layer at a
    regular interval. This helps ensure that the connection is still up and
    working. Once a failure is detected, a reconnection can be attempted.

    The default for qpid_heartbeat was previously 5 seconds. This is pretty
    aggressive and can lead to invalid timeouts occuring. An example is the
    nova-compute service. It may block in native code (such as libvirt)
    under normal circumstances. When this happens, other greenthreads (such
    as the one responsible for sending the heartbeat) can not be scheduled.
    This patch makes the default a less aggressive 1 minute.

    Fix bug 1050661.

    Change-Id: Idce48944050c331dda85e9b10ab6f5b025845c34

Changed in openstack-common:
status: In Progress → Fix Committed
Robert Kukura (rkukura) on 2012-09-14
tags: added: folsom-rc-potential

Fix proposed to branch: stable/folsom
Review: https://review.openstack.org/13006

tags: added: in-stable-folsom

Reviewed: https://review.openstack.org/13006
Committed: http://github.com/openstack/openstack-common/commit/ee442220de44e18ce972db5e8371704dbd6e18be
Submitter: Jenkins
Branch: stable/folsom

commit ee442220de44e18ce972db5e8371704dbd6e18be
Author: Russell Bryant <email address hidden>
Date: Thu Sep 13 16:12:41 2012 -0400

    Change the qpid_heartbeat default to 60 seconds.

    The heartbeat option enables sending a message at the AMQP layer at a
    regular interval. This helps ensure that the connection is still up and
    working. Once a failure is detected, a reconnection can be attempted.

    The default for qpid_heartbeat was previously 5 seconds. This is pretty
    aggressive and can lead to invalid timeouts occuring. An example is the
    nova-compute service. It may block in native code (such as libvirt)
    under normal circumstances. When this happens, other greenthreads (such
    as the one responsible for sending the heartbeat) can not be scheduled.
    This patch makes the default a less aggressive 1 minute.

    Fix bug 1050661.

    Change-Id: Idce48944050c331dda85e9b10ab6f5b025845c34
    (cherry picked from commit 1a9d5c8df5aeee4faf99b54800566e386166836d)

Changed in nova:
milestone: folsom-rc1 → none
milestone: none → folsom-rc1

Fix proposed to branch: master
Review: https://review.openstack.org/13039

Changed in cinder:
status: Confirmed → In Progress
Changed in nova:
status: Confirmed → In Progress

Fix proposed to branch: master
Review: https://review.openstack.org/13041

Changed in quantum:
status: Confirmed → In Progress
Changed in glance:
assignee: nobody → Russell Bryant (russellb)
importance: Undecided → Medium
status: New → Confirmed

Fix proposed to branch: master
Review: https://review.openstack.org/13045

Changed in glance:
status: Confirmed → In Progress
Changed in quantum:
importance: Undecided → High

Reviewed: https://review.openstack.org/13041
Committed: http://github.com/openstack/nova/commit/0c47b74b84a274a7380ace9596b8b1fce62576df
Submitter: Jenkins
Branch: master

commit 0c47b74b84a274a7380ace9596b8b1fce62576df
Author: Russell Bryant <email address hidden>
Date: Fri Sep 14 13:59:03 2012 -0400

    Sync a change to rpc from openstack-common.

    This commit pulls in the following change from openstack-common:

        ee44222 Change the qpid_heartbeat default to 60 seconds.

    Fix bug 1050661.

    Change-Id: I42acdca42f27fbbe9a52441f8cf7ea8261fd8d24

Changed in nova:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/13039
Committed: http://github.com/openstack/cinder/commit/1a408818e96434778d34415e880d81c6f176233f
Submitter: Jenkins
Branch: master

commit 1a408818e96434778d34415e880d81c6f176233f
Author: Russell Bryant <email address hidden>
Date: Fri Sep 14 13:50:47 2012 -0400

    Sync a change to rpc from openstack-common.

    This commit pulls in the following change from openstack-common:

        ee44222 Change the qpid_heartbeat default to 60 seconds.

    Fix bug 1050661.

    Change-Id: I999e16c18227f9aab4dc8c2ff88cd6907c7c8191

Changed in cinder:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/13042
Committed: http://github.com/openstack/quantum/commit/c307e5e26f72e7567e4c6c916a04e281f65fe2a2
Submitter: Jenkins
Branch: master

commit c307e5e26f72e7567e4c6c916a04e281f65fe2a2
Author: Russell Bryant <email address hidden>
Date: Fri Sep 14 14:00:45 2012 -0400

    Sync a change to rpc from openstack-common.

    This commit pulls in the following change from openstack-common:

        ee44222 Change the qpid_heartbeat default to 60 seconds.

    Fix bug 1050661.

    Change-Id: I2cb63d091f3d8a9a35d0e35ef18b70409c1f09ae

Changed in quantum:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/13045
Committed: http://github.com/openstack/glance/commit/185b84ed994a90785819d03efbac2749700c5488
Submitter: Jenkins
Branch: master

commit 185b84ed994a90785819d03efbac2749700c5488
Author: Russell Bryant <email address hidden>
Date: Fri Sep 14 15:08:11 2012 -0400

    Change qpid_heartbeat default.

    The default value of qpid_heartbeat of 5 seconds is unnecessarily
    aggressive. Change it to 60 seconds to reduce the possibility of
    getting disconnected from qpidd under normal circumstances.

    Fix bug 1050661.

    Change-Id: I9525e74f96314c2c9183d6b10cf503936c81f589

Changed in glance:
status: In Progress → Fix Committed
tags: removed: folsom-rc-potential
Thierry Carrez (ttx) on 2012-09-18
Changed in cinder:
status: Fix Committed → Fix Released
Brian Waldon (bcwaldon) on 2012-09-18
Changed in glance:
milestone: none → grizzly-1
tags: added: folsom-rc-potential
no longer affects: glance/folsom
Brian Waldon (bcwaldon) on 2012-09-18
tags: removed: folsom-rc-potential
Thierry Carrez (ttx) on 2012-09-19
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2012-09-19
no longer affects: glance/grizzly
no longer affects: glance/folsom
Changed in glance:
milestone: grizzly-1 → folsom-rc2

Reviewed: https://review.openstack.org/13230
Committed: http://github.com/openstack/glance/commit/c54c92e87aa89ceae97453af73007c5523818abb
Submitter: Jenkins
Branch: milestone-proposed

commit c54c92e87aa89ceae97453af73007c5523818abb
Author: Russell Bryant <email address hidden>
Date: Fri Sep 14 15:08:11 2012 -0400

    Change qpid_heartbeat default.

    The default value of qpid_heartbeat of 5 seconds is unnecessarily
    aggressive. Change it to 60 seconds to reduce the possibility of
    getting disconnected from qpidd under normal circumstances.

    Fix bug 1050661.

    Change-Id: I9525e74f96314c2c9183d6b10cf503936c81f589

Changed in glance:
status: Fix Committed → Fix Released
Robert Kukura (rkukura) on 2012-09-19
Changed in quantum:
milestone: none → folsom-rc2

Reviewed: https://review.openstack.org/13335
Committed: http://github.com/openstack/quantum/commit/6afc60a440a39391e2fc805f24c7b88df11f991f
Submitter: Jenkins
Branch: milestone-proposed

commit 6afc60a440a39391e2fc805f24c7b88df11f991f
Author: Russell Bryant <email address hidden>
Date: Fri Sep 14 14:00:45 2012 -0400

    Sync a change to rpc from openstack-common.

    This commit pulls in the following change from openstack-common:

        ee44222 Change the qpid_heartbeat default to 60 seconds.

    Fix bug 1050661.

    Change-Id: I2cb63d091f3d8a9a35d0e35ef18b70409c1f09ae

Changed in quantum:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2012-09-27
Changed in glance:
milestone: folsom-rc2 → 2012.2
Changed in cinder:
milestone: folsom-rc1 → 2012.2
Thierry Carrez (ttx) on 2012-09-27
Changed in quantum:
milestone: folsom-rc2 → 2012.2
Thierry Carrez (ttx) on 2012-09-27
Changed in nova:
milestone: folsom-rc1 → 2012.2
Mark McLoughlin (markmc) on 2012-11-06
Changed in openstack-common:
milestone: none → 2012.2
status: Fix Committed → Fix Released
Mark McLoughlin (markmc) on 2013-01-22
tags: removed: in-stable-folsom
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers