Zaqar service failed, causing node registration to hang

[Ben Nemec]

I've seen this a number of times in my local dev environments, but it doesn't reproduce consistently. For some reason the zaqar service is in a failed state, which causes the node registration workflow to hang. In the zaqar logs is the following:

2016-11-09 18:38:08.539 19123 INFO zaqar.transport.wsgi.driver [-] Serving on host 9.1.1.1:8888
2016-11-09 20:34:15.923 19123 WARNING keystonemiddleware.auth_token [-] Using the in-process token cache is deprecated as of the 4.2.0 release and may be removed in the 5.0.0 release or the 'O' development cycle. The in-process cache causes inconsistent results and high memory usage. When the feature is removed the auth_token middleware will not cache tokens by default which may result in performance issues. It is recommended to use memcache for the auth_token token cache by setting the memcached_servers option.
2016-11-09 20:34:16.442 19123 CRITICAL zaqar [(None,) 0b0279336e324cb7a4078b93909342f3 829d359286c846f1956f8fb6ff2db39e - - -] [project_id:829d359286c846f1956f8fb6ff2db39e] IOError: [Errno 32] Broken pipe
2016-11-09 20:34:16.442 19123 ERROR zaqar Traceback (most recent call last):
2016-11-09 20:34:16.442 19123 ERROR zaqar File "/usr/bin/zaqar-server", line 10, in <module>
2016-11-09 20:34:16.442 19123 ERROR zaqar sys.exit(run())
2016-11-09 20:34:16.442 19123 ERROR zaqar File "/usr/lib/python2.7/site-packages/zaqar/common/cli.py", line 58, in _wrapper
2016-11-09 20:34:16.442 19123 ERROR zaqar _fail(1, ex)
2016-11-09 20:34:16.442 19123 ERROR zaqar File "/usr/lib/python2.7/site-packages/zaqar/common/cli.py", line 36, in _fail
2016-11-09 20:34:16.442 19123 ERROR zaqar print(ex, file=sys.stderr)
2016-11-09 20:34:16.442 19123 ERROR zaqar IOError: [Errno 32] Broken pipe
2016-11-09 20:34:16.442 19123 ERROR zaqar

One possible cause is that I attempted a deployment without registering nodes. This correctly failed, but maybe the error did something that crashed zaqar?

[Thomas Herve (therve)]

I'll keep bug #1661227 which looks similar.