keystone-bin should properly reload configuration and restart services on SIGHUP

[Abhishek Kekane]

When SIGHUP signal is send to keystone service, it stops the process and while restarting the keystone process, it throws AttributeError: 'Server' object has no attribute 'reset'

Steps to reproduce:
1. Run keystone service as daemon.
2. Send SIGHUP signal to keystone service
   kill -1 <parent_process_id_of_keystone>

[Abhishek Kekane (abhishek-kekane)]

After adding reset method in Server class.

When SIGHUP signal is sent to keystone process, it stops and while restarting the keystone process, it throws error: [Errno 9] Bad file descriptor.

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 346, in fire_timers
    timer()
  File "/usr/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 56, in __call__
    cb(*args, **kw)
  File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
    result = function(*args, **kwargs)
  File "/opt/stack/keystone/keystone/common/environment/eventlet_server.py", line 172, in _run
    debug=False
  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 635, in server
    serv = Server(sock, sock.getsockname(),
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
  File "/usr/lib/python2.7/socket.py", line 170, in _dummy
    raise error(EBADF, 'Bad file descriptor')
error: [Errno 9] Bad file descriptor

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/107482

[David Stanek (dstanek)]

Can you explain in a little more detail how you are running Keystone?

What is the end goal here? I'm guessing that you want reread the configuration like other server daemons. If that's the case I'm not sure this would give you that. Right now the configuration is read with keystone-all is run and I don't know if it's ever reread.

Also you mention restarting the process in the description. As far as I can tell this will restart the server within the process, but doesn't have any effect on the process itself.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/112368

[Tushar Patil (tpatil)]

Hi David:

I think Abhishek meant to say, inside the child processes service is stopped and restarted again when SIGHUP signal is received by the keystone process. Also the reason to delete listen method and move part of the functionality to the __init__ and start method was to take care of the issue you are hitting in your patch.
"NotImplementedError: Can't dup an ssl object".

Launcher restart method has the logic to reload the configuration files which gets called when you send SIGHUP signal to the keystone process.

Will appreciate if you can give your feedback on this patch https://review.openstack.org/107482 and Abhishek can take care of it.

Thanks.

[OpenStack Infra (hudson-openstack)]

Change abandoned by David Stanek (<email address hidden>) on branch: master
Review: https://review.openstack.org/112368
Reason: This was meant to show the idea because it was not easy to communicate it over IRC.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/107482
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=825f3e7820423e3accae413070f88bd12e6c9b57
Submitter: Jenkins
Branch: master

commit 825f3e7820423e3accae413070f88bd12e6c9b57
Author: abhishekkekane <email address hidden>
Date: Fri Jul 4 06:41:12 2014 -0700

    Keystone service throws error on receiving SIGHUP

    This patch resolves following erorrs:
    1. AttributeError: 'Server' object has no attribute 'reset'.
    2. error: [Errno 9] Bad file descriptor
    3. Can't dup an SSL object

    When the SIGHUP signal is received by the service launcher in
    common service framework, it calls the server's reset method.
    As reset method is not present in Sever class of
    keystone.common.environment.eventlet_server module, it raises
    AttributeError: 'Server' object has no attribute 'reset'.

    After adding reset method when SIGHUP signal is sent to service
    parent process, it stops the service and then calls service start
    method again. When it stops the service, it kills the eventlet
    thread, which internally closes the wsgi server socket object.
    This server socket object is now not usable again and it throws
    following error, while restarting the service:

    error: [Errno 9] Bad file descriptor

    To resolve 'Bad file descriptor' error, creating duplicate
    socket object, every time service starts.

    As SSL object can not be duplicated, creating duplicate
    socket object before converting a regular socket into an
    SSL socket.

    Closes-Bug: #1337850
    Change-Id: I52caacc01a94428e4986ef68d032ad317e09b276