stopping sahara-engine failed

[Sergey Reshetnyak]

Stopping of sahara-engine failed with exit code 142

Steps to reproduce:
1. Launch sahara:
$ sahara-engine --config-file etc/sahara/sahara.conf

2. Kill engine:
$ killall sahara-engine

Expected result:
sahara-engine exited with code 0

Actual result:

2016-02-16 17:31:40.382 INFO oslo_service.service Caught SIGTERM, stopping children
2016-02-16 17:31:40.382 DEBUG oslo_concurrency.lockutils Acquired semaphore "singleton_lock" from (pid=16164) lock /home/serg/projects/sahara/.tox/venv/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:212
2016-02-16 17:31:40.383 DEBUG oslo_concurrency.lockutils Releasing semaphore "singleton_lock" from (pid=16164) lock /home/serg/projects/sahara/.tox/venv/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:225
2016-02-16 17:31:40.383 DEBUG oslo_service.service Stop services. from (pid=16164) stop /home/serg/projects/sahara/.tox/venv/lib/python2.7/site-packages/oslo_service/service.py:583
2016-02-16 17:31:52.064 DEBUG amqp Closed channel #1 from (pid=16173) _do_close /home/serg/projects/sahara/.tox/venv/lib/python2.7/site-packages/amqp/channel.py:106
2016-02-16 17:31:52.065 DEBUG oslo_concurrency.lockutils Acquired semaphore "singleton_lock" from (pid=16173) lock /home/serg/projects/sahara/.tox/venv/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:212
2016-02-16 17:31:52.065 DEBUG oslo_concurrency.lockutils Releasing semaphore "singleton_lock" from (pid=16173) lock /home/serg/projects/sahara/.tox/venv/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:225
2016-02-16 17:32:10.384 WARNING oslo_messaging.server Possible hang: stop is waiting for start to complete
2016-02-16 17:32:22.165 DEBUG oslo_service.periodic_task Running periodic task SaharaPeriodicTasks.update_job_statuses from (pid=16164) run_periodic_tasks /home/serg/projects/sahara/.tox/venv/lib/python2.7/site-packages/oslo_service/periodic_task.py:215
2016-02-16 17:32:22.165 DEBUG sahara.service.periodic [req-4145708e-95f3-4bcf-b21e-7dd11a855ed5 None None] Updating job statuses from (pid=16164) update_job_statuses /home/serg/projects/sahara/sahara/service/periodic.py:136
Alarm clock

(venv)serg@sreshetniak ~/projects/sahara $ echo $?
142

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/280741

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/280741
Committed: https://git.openstack.org/cgit/openstack/sahara/commit/?id=52b3cccc1ea81f733f7e3d2aa30f3c86c4656aeb
Submitter: Jenkins
Branch: master

commit 52b3cccc1ea81f733f7e3d2aa30f3c86c4656aeb
Author: Sergey Reshetnyak <email address hidden>
Date: Tue Feb 16 17:51:26 2016 +0300

    Start RPC service before waiting

    Change-Id: I9270229e69c48d797bbd79cf351da3c2cc4b2bee
    Closes-bug: #1546119

[Adam Spiers (adam.spiers)]

Thanks for the fix. I saw something nearly identical in Barbican, and porting the same fix over seemed to work:

https://bugs.launchpad.net/barbican/+bug/1705543

[Adam Spiers (adam.spiers)]

FYI there's an ongoing debate on what the correct fix should be for this:

http://lists.openstack.org/pipermail/openstack-dev/2017-September/122243.html