keystone fails to start on icehouse side of grenade, "error: [Errno 98] Address already in use"

[Matt Riedemann]

I know we used to have a bug for this but I can't find it anymore, so opening a new one as this failed a job:

http://logs.openstack.org/84/123784/1/check/check-grenade-dsvm/0082b12/logs/old/screen-key.txt.gz#_2014-09-24_17_42_14_365

2014-09-24 17:42:14.365 32570 ERROR root [-] Failed to start the admin server
2014-09-24 17:42:14.365 32570 TRACE root Traceback (most recent call last):
2014-09-24 17:42:14.365 32570 TRACE root File "/opt/stack/old/keystone/bin/keystone-all", line 79, in serve
2014-09-24 17:42:14.365 32570 TRACE root server.start()
2014-09-24 17:42:14.365 32570 TRACE root File "/opt/stack/old/keystone/keystone/common/environment/eventlet_server.py", line 89, in start
2014-09-24 17:42:14.365 32570 TRACE root backlog=backlog)
2014-09-24 17:42:14.365 32570 TRACE root File "/usr/lib/python2.7/dist-packages/eventlet/convenience.py", line 38, in listen
2014-09-24 17:42:14.365 32570 TRACE root sock.bind(addr)
2014-09-24 17:42:14.365 32570 TRACE root File "/usr/lib/python2.7/socket.py", line 224, in meth
2014-09-24 17:42:14.365 32570 TRACE root return getattr(self._sock,name)(*args)
2014-09-24 17:42:14.365 32570 TRACE root error: [Errno 98] Address already in use
2014-09-24 17:42:14.365 32570 TRACE root

[Matt Riedemann (mriedem)]

message:"Failed to start the admin server" AND message:"Address already in use" AND tags:"screen-key.txt"

1 hit in 7 days but it was in the gate (and it wasn't the one above that was in the check queue).

There are 2 hits in 10 days.

[Matt Riedemann (mriedem)]

e-r query: https://review.openstack.org/#/c/123910/

[Matt Riedemann (mriedem)]

Looks like this is probably due to the fact that we're still running keystone with eventlet in Icehouse:

http://logs.openstack.org/84/123784/1/check/check-grenade-dsvm/0082b12/logs/old/screen-key.txt.gz#_2014-09-24_17_42_14_013

The thinking is we're not seeing this issue anymore in Juno because we're running with Apache in the gate.

[Morgan Fainberg (mdrnstm)]

I'd keep my eye on the postgres job (Master/Juno) as that is also under Eventlet. It seems fairly uncommon but i'll keep my eyes on the occurrence so we can keep it on the radar.