Designate service_statuses is not working

[Takahito Hirose]

When I set the 2 or more each designate service workers, those service was not working.
At first, I found this problem on newton version. but it happened this problem newer version better then newton.

I set the number of worker is 5 and set the same as logging point.

2019-04-04 12:37:24.512 98588 DEBUG designate.service_status [-] designate/service_status.HeartBeatEmitter.__init__: 98588 __init__ /usr/lib/python2.7/site-packages/designate/service_status.py:53
2019-04-04 12:37:24.533 98598 DEBUG designate.service_status [-] designate/service_status.HeartBeatEmitter.start: 98598 start /usr/lib/python2.7/site-packages/designate/service_status.py:100
2019-04-04 12:37:24.533 98598 DEBUG designate.service_status [-] designate/service_status.HeartBeatEmitter.start: _running=True start /usr/lib/python2.7/site-packages/designate/service_status.py:102
2019-04-04 12:37:24.536 98599 DEBUG designate.service_status [-] designate/service_status.HeartBeatEmitter.start: 98599 start /usr/lib/python2.7/site-packages/designate/service_status.py:100
2019-04-04 12:37:24.536 98599 DEBUG designate.service_status [-] designate/service_status.HeartBeatEmitter.start: _running=True start /usr/lib/python2.7/site-packages/designate/service_status.py:102
2019-04-04 12:37:24.537 98600 DEBUG designate.service_status [-] designate/service_status.HeartBeatEmitter.start: 98600 start /usr/lib/python2.7/site-packages/designate/service_status.py:100
2019-04-04 12:37:24.538 98600 DEBUG designate.service_status [-] designate/service_status.HeartBeatEmitter.start: _running=True start /usr/lib/python2.7/site-packages/designate/service_status.py:102
2019-04-04 12:37:24.563 98601 DEBUG designate.service_status [-] designate/service_status.HeartBeatEmitter.start: 98601 start /usr/lib/python2.7/site-packages/designate/service_status.py:100
2019-04-04 12:37:24.563 98601 DEBUG designate.service_status [-] designate/service_status.HeartBeatEmitter.start: _running=True start /usr/lib/python2.7/site-packages/designate/service_status.py:102
2019-04-04 12:37:24.636 98588 DEBUG designate.service_status [-] designate/service_status.HeartBeatEmitter._emit_heartbeat: 98588 _emit_heartbeat /usr/lib/python2.7/site-packages/designate/service_status.py:75
2019-04-04 12:37:24.636 98588 DEBUG designate.service_status [-] designate/service_status.HeartBeatEmitter._emit_heartbeat: _running=False _emit_heartbeat /usr/lib/python2.7/site-packages/designate/service_status.py:76
2019-04-04 12:37:29.637 98588 DEBUG designate.service_status [-] designate/service_status.HeartBeatEmitter._emit_heartbeat: 98588 _emit_heartbeat /usr/lib/python2.7/site-packages/designate/service_status.py:75
2019-04-04 12:37:29.637 98588 DEBUG designate.service_status [-] designate/service_status.HeartBeatEmitter._emit_heartbeat: _running=False _emit_heartbeat /usr/lib/python2.7/site-packages/designate/service_status.py:76
__init__ and _emit_heartbeat are same process id. But start is not same as those.

__init__ , _emit_heartbeat: 98588

start : 98598, 98598, 98599, 98600, 98601

then I checked concern about process IDs.

# ps auxwwf | grep designate-api
designa+ 98588 1.9 0.7 347336 62380 ? Ss 12:37 0:02 /usr/bin/python2 /usr/bin/designate-api --config-file /etc/designate/designate.conf --log-file /var/log/designate/api.log
designa+ 98598 0.2 0.8 392348 69792 ? S 12:37 0:00 \_ /usr/bin/python2 /usr/bin/designate-api --config-file /etc/designate/designate.conf --log-file /var/log/designate/api.log
designa+ 98599 0.2 0.8 392344 69784 ? S 12:37 0:00 \_ /usr/bin/python2 /usr/bin/designate-api --config-file /etc/designate/designate.conf --log-file /var/log/designate/api.log
designa+ 98600 0.2 0.8 392344 69784 ? S 12:37 0:00 \_ /usr/bin/python2 /usr/bin/designate-api --config-file /etc/designate/designate.conf --log-file /var/log/designate/api.log
designa+ 98601 0.2 0.8 392348 69784 ? S 12:37 0:00 \_ /usr/bin/python2 /usr/bin/designate-api --config-file /etc/designate/designate.conf --log-file /var/log/designate/api.log
98588 is parent process ID, but others are child process.

So if _running flag update to True, It never start the _emit_heartbeat.

How do I fix this problem
# diff -u /usr/lib/python2.7/site-packages/designate/cmd/api.py.upstream/usr/lib/python2.7/site-packages/designate/cmd/api.py
--- /usr/lib/python2.7/site-packages/designate/cmd/api.py.upstream 2019-04-03 18:10:09.908918965 +0900
+++ /usr/lib/python2.7/site-packages/designate/cmd/api.py 2019-04-03 18:09:49.886038819 +0900
@@ -40,4 +40,5 @@

     server = api_service.Service(threads=CONF['service:api'].threads)
     service.serve(server, workers=CONF['service:api'].workers)
+ server.heartbeat_emitter.start()
     service.wait()

I fixed to call the start method from parent process. I think all process have same problem. So I think It is need to fix same way.

But I couldn't know how do I write the unittest in this case. Do you guys have any idea? I couldn't find the multi workers case in designate's unittest.

reference and detail: https://<email address hidden>/when-increasing-the-designate-worker-it-doesnt-work-the-service-statuses-6a4372d8f3e0

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/657382

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/657382
Committed: https://git.openstack.org/cgit/openstack/designate/commit/?id=9214a42a8e502b135224d5f7d0f3871ac7daa00f
Submitter: Zuul
Branch: master

commit 9214a42a8e502b135224d5f7d0f3871ac7daa00f
Author: Takahito Hirose <email address hidden>
Date: Mon May 6 21:47:33 2019 +0900

    Designate service_statuses is not working

    When I set the 2 or more each designate service workers,
    those service_status was not working.
    In this case, The 'HeartBeatEmitter.start' is called
    by child processes.
    And The 'HeartBeatEmitter._emit_heartbeat' is called
    by Parent process.
    Parent process' '_running' value never change to TRUE
    on parent process.
    So in this case, designate isn't emitted the heartbeat.

    Change-Id: I1338d2ca180274ae84d4b358ff467e6655eb8022
    Closes-Bug: #1827070

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/designate 9.0.0.0rc1 release candidate.