rabbitmq-server amulet test timeout inadequate while system is under heavy load

[Ryan Beisner]

The rabbitmq-server amulet test timeout inadequate while system is under heavy load.

Specifically, the self.d.sentry.wait() default timeout is 300s (5m). That is not always long enough.

[Ryan Beisner (1chb1n)]

One example:

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_amulet_full/openstack/charm-rabbitmq-server/389380/1/322/consoleText.test_charm_amulet_full_1084.txt

DEBUG:runner:Traceback (most recent call last):
DEBUG:runner: File "/tmp/bundletester-v_Vicy/rabbitmq-server/tests/gate-basic-trusty-kilo", line 25, in <module>
DEBUG:runner: deployment.run_tests()
DEBUG:runner: File "/tmp/bundletester-v_Vicy/rabbitmq-server/tests/charmhelpers/contrib/amulet/deployment.py", line 97, in run_tests
DEBUG:runner: getattr(self, test)()
DEBUG:runner: File "/tmp/bundletester-v_Vicy/rabbitmq-server/tests/basic_deployment.py", line 510, in test_412_rmq_management_plugin
DEBUG:runner: self.d.sentry.wait()
DEBUG:runner: File "/var/lib/jenkins/checkout/0/rabbitmq-server/.tox/func27/local/lib/python2.7/site-packages/amulet/sentry.py", line 541, in wait
DEBUG:runner: for i in helpers.timeout_gen(timeout):
DEBUG:runner: File "/var/lib/jenkins/checkout/0/rabbitmq-server/.tox/func27/local/lib/python2.7/site-packages/amulet/helpers.py", line 109, in timeout_gen
DEBUG:runner: raise TimeoutError()
DEBUG:runner:amulet.helpers.TimeoutError
DEBUG:runner:Exit Code: 1

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/391261
Committed: https://git.openstack.org/cgit/openstack/charm-rabbitmq-server/commit/?id=0ee37a56e58956e2bc71277c50ba440b1bb3ac3d
Submitter: Jenkins
Branch: master

commit 0ee37a56e58956e2bc71277c50ba440b1bb3ac3d
Author: Ryan Beisner <email address hidden>
Date: Thu Oct 27 15:46:25 2016 +0000

    Set Amulet wait timeouts

    The Amulet sentry.wait() default of 300 seconds is sometimes not
    long enough, causing false test failures.

    Closes-Bug: #1637960

    Change-Id: I9e2677b5fabe16e86e82d185662f86a63f4cd0d4

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ryan Beisner (<email address hidden>) on branch: stable/16.10
Review: https://review.openstack.org/391264

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/391264
Committed: https://git.openstack.org/cgit/openstack/charm-rabbitmq-server/commit/?id=19ffe12cb169fade4a73070bfccf6b0c2b1db041
Submitter: Jenkins
Branch: stable/16.10

commit 19ffe12cb169fade4a73070bfccf6b0c2b1db041
Author: Ryan Beisner <email address hidden>
Date: Thu Oct 27 15:46:25 2016 +0000

    Set Amulet wait timeouts

    The Amulet sentry.wait() default of 300 seconds is sometimes not
    long enough, causing false test failures.

    Closes-Bug: #1637960

    Change-Id: I9e2677b5fabe16e86e82d185662f86a63f4cd0d4

[Ryan Beisner (1chb1n)]

Fix landed for stable/16.10 release.