In another reproduction (with additional timestamp debug bits added to the test) of this errored pause test, here is some interesting timing info:
## Timestamps from tests
2016-05-12 20:11:24,131 test_910_pause_and_resume DEBUG: Checking pause and resume actions...
2016-05-12 20:11:24,134 test_910_pause_and_resume INFO: Waiting on extended status checks...
2016-05-12 20:11:24,135 _auto_wait_for_status INFO: Waiting for extended status on units...
2016-05-12 20:11:24,137 _auto_wait_for_status DEBUG: Default extended status wait match: contains READY (case-insensitive)
2016-05-12 20:11:24,137 _auto_wait_for_status DEBUG: Excluding services from extended status match: ['mysql']
2016-05-12 20:11:24,138 _auto_wait_for_status DEBUG: Waiting up to 1800s for extended status on services: ['cinder', 'glance', 'keystone', 'rabbitmq-server']
2016-05-12 20:11:24,727 _auto_wait_for_status INFO: OK
2016-05-12 20:11:24,728 test_910_pause_and_resume DEBUG: Unit name: cinder/6
2016-05-12 20:11:24,729 test_910_pause_and_resume DEBUG: Checking for active status on cinder/6
2016-05-12 20:11:38,170 test_910_pause_and_resume DEBUG: Running pause action on cinder/6
Running command: juju action do --format=json cinder/6 pause
2016-05-12 20:11:38,998 test_910_pause_and_resume DEBUG: Waiting on action 1f3e7042-bbb7-4025-82d8-1ca3f6254653
Traceback (most recent call last):
File "/tmp/bundletester-V7qqmq/cinder/tests/gate-basic-trusty-liberty", line 11, in <module>
deployment.run_tests()
File "/tmp/bundletester-V7qqmq/cinder/tests/charmhelpers/contrib/amulet/deployment.py", line 95, in run_tests
getattr(self, test)()
File "/tmp/bundletester-V7qqmq/cinder/tests/basic_deployment.py", line 780, in test_910_pause_and_resume
"Pause action failed at {}".format(datetime.datetime.now())
AssertionError: Pause action failed at 2016-05-12 20:11:49.244020
## SIGTERM Timestamps from cinder logs
ubuntu@juju-beis0-machine-31:/var/log/cinder$ grep SIG *
cinder-api.log:2016-05-12 20:03:07.062 14524 INFO oslo_service.service [-] Caught SIGTERM, stopping children
cinder-api.log:2016-05-12 20:12:21.951 32034 INFO oslo_service.service [-] Caught SIGTERM, stopping children
cinder-scheduler.log:2016-05-12 20:03:07.454 14576 INFO oslo_service.service [req-42bbdc02-be8c-437c-a087-5006710eef11 - - - - -] Caught SIGTERM, exiting
cinder-scheduler.log:2016-05-12 20:12:22.396 32089 INFO oslo_service.service [req-70ae4130-3d76-46a9-98a6-91a14fa47afa - - - - -] Caught SIGTERM, exiting
cinder-volume.log:2016-05-12 20:03:07.216 14630 INFO oslo_service.service [req-9f77de11-e5a6-42d5-a436-7cb2c900fd29 - - - - -] Caught SIGTERM, stopping children
cinder-volume.log:2016-05-12 20:12:22.118 32058 INFO oslo_service.service [req-c43b791d-e062-4148-b23c-5cdfe55ebb79 - - - - -] Caught SIGTERM, stopping children
## To summarize:
20:11:38 Test issues pause action
20:11:49 Test declares failure (based on juju action fetch returning non-zero)
20:12:22 Services are stopped
In another reproduction (with additional timestamp debug bits added to the test) of this errored pause test, here is some interesting timing info:
## Timestamps from tests pause_and_ resume DEBUG: Checking pause and resume actions... pause_and_ resume INFO: Waiting on extended status checks... for_status INFO: Waiting for extended status on units... for_status DEBUG: Default extended status wait match: contains READY (case-insensitive) for_status DEBUG: Excluding services from extended status match: ['mysql'] for_status DEBUG: Waiting up to 1800s for extended status on services: ['cinder', 'glance', 'keystone', 'rabbitmq-server'] for_status INFO: OK pause_and_ resume DEBUG: Unit name: cinder/6 pause_and_ resume DEBUG: Checking for active status on cinder/6 pause_and_ resume DEBUG: Running pause action on cinder/6
2016-05-12 20:11:24,131 test_910_
2016-05-12 20:11:24,134 test_910_
2016-05-12 20:11:24,135 _auto_wait_
2016-05-12 20:11:24,137 _auto_wait_
2016-05-12 20:11:24,137 _auto_wait_
2016-05-12 20:11:24,138 _auto_wait_
2016-05-12 20:11:24,727 _auto_wait_
2016-05-12 20:11:24,728 test_910_
2016-05-12 20:11:24,729 test_910_
2016-05-12 20:11:38,170 test_910_
Running command: juju action do --format=json cinder/6 pause
2016-05-12 20:11:38,998 test_910_ pause_and_ resume DEBUG: Waiting on action 1f3e7042- bbb7-4025- 82d8-1ca3f62546 53 ter-V7qqmq/ cinder/ tests/gate- basic-trusty- liberty" , line 11, in <module> run_tests( ) ter-V7qqmq/ cinder/ tests/charmhelp ers/contrib/ amulet/ deployment. py", line 95, in run_tests ter-V7qqmq/ cinder/ tests/basic_ deployment. py", line 780, in test_910_ pause_and_ resume datetime. datetime. now())
Traceback (most recent call last):
File "/tmp/bundletes
deployment.
File "/tmp/bundletes
getattr(self, test)()
File "/tmp/bundletes
"Pause action failed at {}".format(
AssertionError: Pause action failed at 2016-05-12 20:11:49.244020
## SIGTERM Timestamps from cinder logs juju-beis0- machine- 31:/var/ log/cinder$ grep SIG * api.log: 2016-05- 12 20:03:07.062 14524 INFO oslo_service. service [-] Caught SIGTERM, stopping children api.log: 2016-05- 12 20:12:21.951 32034 INFO oslo_service. service [-] Caught SIGTERM, stopping children scheduler. log:2016- 05-12 20:03:07.454 14576 INFO oslo_service. service [req-42bbdc02- be8c-437c- a087-5006710eef 11 - - - - -] Caught SIGTERM, exiting scheduler. log:2016- 05-12 20:12:22.396 32089 INFO oslo_service. service [req-70ae4130- 3d76-46a9- 98a6-91a14fa47a fa - - - - -] Caught SIGTERM, exiting volume. log:2016- 05-12 20:03:07.216 14630 INFO oslo_service. service [req-9f77de11- e5a6-42d5- a436-7cb2c900fd 29 - - - - -] Caught SIGTERM, stopping children volume. log:2016- 05-12 20:12:22.118 32058 INFO oslo_service. service [req-c43b791d- e062-4148- b23c-5cdfe55ebb 79 - - - - -] Caught SIGTERM, stopping children
ubuntu@
cinder-
cinder-
cinder-
cinder-
cinder-
cinder-
ubuntu@ juju-beis0- machine- 31:/var/ log$ ps aux | grep cinder juju/tools/ unit-cinder- 6/jujud unit --data-dir /var/lib/juju --unit-name cinder/6 --debug juju-beis0- machine- 31:/var/ log$ ps aux | grep oslo
root 3396 0.2 1.5 349744 24344 ? Ssl 20:01 0:02 /var/lib/
ubuntu 4517 0.0 0.0 10460 936 pts/0 S+ 20:21 0:00 grep --color=auto cinder
ubuntu@
ubuntu 4519 0.0 0.0 10460 932 pts/0 S+ 20:21 0:00 grep --color=auto oslo
## To summarize:
20:11:38 Test issues pause action
20:11:49 Test declares failure (based on juju action fetch returning non-zero)
20:12:22 Services are stopped