OpenStack Compute (nova)

Nova api service outputs error messages when SIGHUP signal is sent

Bug #1334651 reported by Rajesh Tailor
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Rajesh Tailor
OpenStack Compute (nova) 2014.2 "juno"
oslo-incubator
Invalid
Medium
Unassigned

Bug Description

When SIGHUP signal is send to nova-api service, it stops all the nova-api processes and while restarting the nova-api processes, it throws AttributeError: 'WSGIService' object has no attribute 'reset'.

2014-06-24 15:52:55.185 CRITICAL nova [-] AttributeError: 'WSGIService' object has no attribute 'reset'

2014-06-24 15:52:55.185 TRACE nova Traceback (most recent call last):
2014-06-24 15:52:55.185 TRACE nova File "/usr/local/bin/nova-api", line 10, in <module>
2014-06-24 15:52:55.185 TRACE nova sys.exit(main())
2014-06-24 15:52:55.185 TRACE nova File "/opt/stack/nova/nova/cmd/api.py", line 56, in main
2014-06-24 15:52:55.185 TRACE nova launcher.launch_service(server, workers=server.workers or 1)
2014-06-24 15:52:55.185 TRACE nova File "/opt/stack/nova/nova/openstack/common/service.py", line 340, in launch_service
2014-06-24 15:52:55.185 TRACE nova self._start_child(wrap)
2014-06-24 15:52:55.185 TRACE nova File "/opt/stack/nova/nova/openstack/common/service.py", line 324, in _start_child
2014-06-24 15:52:55.185 TRACE nova launcher.restart()
2014-06-24 15:52:55.185 TRACE nova File "/opt/stack/nova/nova/openstack/common/service.py", line 145, in restart
2014-06-24 15:52:55.185 TRACE nova self.services.restart()
2014-06-24 15:52:55.185 TRACE nova File "/opt/stack/nova/nova/openstack/common/service.py", line 478, in restart
2014-06-24 15:52:55.185 TRACE nova restart_service.reset()
2014-06-24 15:52:55.185 TRACE nova AttributeError: 'WSGIService' object has no attribute 'reset'
2014-06-24 15:52:55.185 TRACE nova

Steps to reproduce:
1. Run nova-api service as daemon.
2. Send SIGHUP signal to nova-api service
   kill -1 <parent_process_id_of_nova_api>

Tags: ntt
Rajesh Tailor (rajesh-tailor)
Changed in nova:
assignee: nobody → Rajesh Tailor (rajesh-tailor)
Revision history for this message
Rajesh Tailor (rajesh-tailor) wrote :
Download full text (4.2 KiB)

After adding reset method in WSGIService class.
When SIGHUP signal is send to nova-api service, it stops all the nova-api processes
and while restarting the nova-api processes, it throws
error: [Errno 9] Bad file descriptor.

File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 56, in __call__
    cb(*args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
    result = function(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 635, in server
Traceback (most recent call last):
Traceback (most recent call last):
    serv = Server(sock, sock.getsockname(),
  File "/usr/lib/python2.7/socket.py", line 224, in meth
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
2014-06-26 20:38:34.240 INFO nova.wsgi [-] WSGI server has stopped.
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 346, in fire_timers
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 56, in __call__
    cb(*args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
    timer()
    result = function(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 635, in server
    timer()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 56, in __call__
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 56, in __call__
    cb(*args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
    serv = Server(sock, sock.getsockname(),
2014-06-26 20:38:34.241 INFO nova.wsgi [-] WSGI server has stopped.
    cb(*args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
2014-06-26 20:38:34.241 INFO nova.wsgi [-] WSGI server has stopped.
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 346, in fire_timers
    result = function(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 56, in __call__
  File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 635, in server
Traceback (most recent call last):
    result = function(*args, **kwargs)
    return getattr(self._sock,name)(*args)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 635, in server
Traceback (most recent call last):
    result = function(*args, **kwargs)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 346, in fire_timers
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 346, in fire_timers
    serv = Server(sock, sock.getsockname(),
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    timer()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 56, in __call__
    timer()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 56, in __call__
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    timer()
    return getattr(self._sock,name)(*args)
  File "/usr/local/lib/python2.7/dist-pac...

Read more...

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/104887

Changed in nova:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/104887
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2f3d774eb51eac9a370813362047a2cb3d124d86
Submitter: Jenkins
Branch: master

commit 2f3d774eb51eac9a370813362047a2cb3d124d86
Author: Rajesh Tailor <email address hidden>
Date: Thu Jul 3 07:44:20 2014 -0700

    Nova-api service throws error when SIGHUP is sent

    Added reset method in WSGIService class.

    After adding reset method when SIGHUP signal is sent to
    wsgi service parent process,then it sends SIGHUP signal
    to all of its child processes. Each child process handles
    SIGHUP signal by first stopping the service and then calls
    service start method again. When it stops the service, it
    kills the eventlet thread, which internally closes the wsgi
    server socket object. This server socket object is now not
    usable again and it throws following error, while restarting
    the service:

    error: [Errno 9] Bad file descriptor

    To resolve 'Bad file descriptor' error, creating duplicate
    socket object, every time service starts.

    When the wsgi service is stopped, it sets the green pool
    size to 0, so resizing the green pool to default value,
    on service restart.

    Closes-Bug: #1334651

    Change-Id: If1035deaf8b31f2712d88f0112fdd2e9e9dc7cb0

Changed in nova:
status: In Progress → Fix Committed
Russell Bryant (russellb)
Changed in nova:
milestone: none → juno-2
status: Fix Committed → Fix Released
Doug Hellmann (doug-hellmann)
Changed in oslo:
importance: Undecided → Medium
status: New → Triaged
Thierry Carrez (ttx)
Changed in nova:
milestone: juno-2 → 2014.2
Revision history for this message
Tiantian Gao (gtt116) wrote :

Hi Rajesh,

Does nova-api really support SIGHUP? I test it in devstack environment, the last commit of nova is a7e1433a79b30937e29f69a866172c061ced5a5f(Thu Oct 30 00:35:25 2014 +0000).

When I hit "kill -1 $PID_OF_NOVA_API_PARENT" nova-api just log below messages then exit
2014-11-04 08:28:38.520 8249 INFO nova.openstack.common.service [-] Caught SIGHUP, stopping children
2014-11-04 08:28:38.521 8332 INFO nova.openstack.common.service [-] Child caught SIGTERM, exiting
2014-11-04 08:28:38.521 8294 INFO nova.openstack.common.service [-] Child caught SIGTERM, exiting
....
2014-11-04 08:28:38.557 8249 INFO nova.openstack.common.service [-] Child 8294 exited with status 1
2014-11-04 08:28:38.557 8249 INFO nova.openstack.common.service [-] Child 8295 exited with status 1
2014-11-04 08:28:38.557 8249 INFO nova.openstack.common.service [-] Child 8296 exited with status 1
2014-11-04 08:28:38.558 8249 INFO nova.openstack.common.service [-] Child 8322 exited with status 1

But in your environment I believe nova plays very well with SIGHUP, is there some config items to enable this feature? do I missing some thing?

Revision history for this message
Rajesh Tailor (rajesh-tailor) wrote :

Hi Tiantian,

You need to run the nova-api service as daemon.

Steps to reproduce:
1. Run nova-api service as daemon.
2. Send SIGHUP signal to nova-api service
   kill -1 <parent_process_id_of_nova_api>

Just for testing in your environment, you can use below work around to fulfill step 1.

diff --git a/nova/openstack/common/service.py b/nova/openstack/common/servic
index f682b2b..12127d6 100644
--- a/nova/openstack/common/service.py
+++ b/nova/openstack/common/service.py
@@ -77,7 +77,7 @@ def _is_sighup_and_daemon(signo):
         # Avoid checking if we are a daemon, because the signal isn't
         # SIGHUP.
         return False
- return _is_daemon()
+ return True

 def _signo_to_signame(signo):

Revision history for this message
Tiantian Gao (gtt116) wrote : Re: [Bug 1334651] Re: Nova api service outputs error messages when SIGHUP signal is sent
Download full text (3.1 KiB)

Hi Rajesh,

Thank you very much for the patch. It works well for me.

2014-11-04 19:00 GMT+08:00 Rajesh Tailor <email address hidden>:

> Hi Tiantian,
>
> You need to run the nova-api service as daemon.
>
> Steps to reproduce:
> 1. Run nova-api service as daemon.
> 2. Send SIGHUP signal to nova-api service
> kill -1 <parent_process_id_of_nova_api>
>
> Just for testing in your environment, you can use below work around to
> fulfill step 1.
>
> diff --git a/nova/openstack/common/service.py
> b/nova/openstack/common/servic
> index f682b2b..12127d6 100644
> --- a/nova/openstack/common/service.py
> +++ b/nova/openstack/common/service.py
> @@ -77,7 +77,7 @@ def _is_sighup_and_daemon(signo):
> # Avoid checking if we are a daemon, because the signal isn't
> # SIGHUP.
> return False
> - return _is_daemon()
> + return True
>
> def _signo_to_signame(signo):
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1334651
>
> Title:
> Nova api service outputs error messages when SIGHUP signal is sent
>
> Status in OpenStack Compute (Nova):
> Fix Released
> Status in The Oslo library incubator:
> Triaged
>
> Bug description:
> When SIGHUP signal is send to nova-api service, it stops all the nova-
> api processes and while restarting the nova-api processes, it throws
> AttributeError: 'WSGIService' object has no attribute 'reset'.
>
>
> 2014-06-24 15:52:55.185 CRITICAL nova [-] AttributeError: 'WSGIService'
> object has no attribute 'reset'
>
> 2014-06-24 15:52:55.185 TRACE nova Traceback (most recent call last):
> 2014-06-24 15:52:55.185 TRACE nova File "/usr/local/bin/nova-api",
> line 10, in <module>
> 2014-06-24 15:52:55.185 TRACE nova sys.exit(main())
> 2014-06-24 15:52:55.185 TRACE nova File
> "/opt/stack/nova/nova/cmd/api.py", line 56, in main
> 2014-06-24 15:52:55.185 TRACE nova launcher.launch_service(server,
> workers=server.workers or 1)
> 2014-06-24 15:52:55.185 TRACE nova File
> "/opt/stack/nova/nova/openstack/common/service.py", line 340, in
> launch_service
> 2014-06-24 15:52:55.185 TRACE nova self._start_child(wrap)
> 2014-06-24 15:52:55.185 TRACE nova File
> "/opt/stack/nova/nova/openstack/common/service.py", line 324, in
> _start_child
> 2014-06-24 15:52:55.185 TRACE nova launcher.restart()
> 2014-06-24 15:52:55.185 TRACE nova File
> "/opt/stack/nova/nova/openstack/common/service.py", line 145, in restart
> 2014-06-24 15:52:55.185 TRACE nova self.services.restart()
> 2014-06-24 15:52:55.185 TRACE nova File
> "/opt/stack/nova/nova/openstack/common/service.py", line 478, in restart
> 2014-06-24 15:52:55.185 TRACE nova restart_service.reset()
> 2014-06-24 15:52:55.185 TRACE nova AttributeError: 'WSGIService' object
> has no attribute 'reset'
> 2014-06-24 15:52:55.185 TRACE nova
>
> Steps to reproduce:
> 1. Run nova-api service as daemon.
> 2. Send SIGHUP signal to nova-api service
> kill -1 <parent_process_id_of_nova_api>
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nova/+bug/1334651/+subscriptions
>

--
Be...

Read more...

Revision history for this message
Ben Nemec (bnemec) wrote :

It appears this was fixed in Nova and there's nothing to be done in Oslo. Let me know if I'm wrong about that.

Changed in oslo-incubator:
status: Triaged → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.