Master node deployment failed on building keystone container withCRITICAL keystone [-] error: [Errno 98] Address already in use, Could not bind to 0.0.0.0:35357

[Andrey Sledzinskiy]

Reproduced on 7.0-301 iso with MU1

Steps:
1. Install fuel
2. Ssh to master and add new repo to /etc/yum.repos.d/ with next data:
[temporary-0]
name=temporary-0
baseurl=http://osci-mirror-msk.msk.mirantis.net/mos-repos/centos/mos7.0-centos6-fuel/snapshots/proposed-latest/x86_64/
gpgcheck=0
priority=1
3. Execute on master node
- yum clean expire-cache; yum update -y
- dockerctl destroy all; bootstrap_admin_node.sh

Actual result - master node re-deployment failed on building keystone container

from /var/log/docker-logs/keystone/keystone-all.log:
2015-11-08 12:57:27.892 786 ERROR keystone.common.environment.eventlet_server [-] Could not bind to 0.0.0.0:35357
2015-11-08 12:57:27.893 786 ERROR root [-] Failed to start the admin server
2015-11-08 12:57:27.893 786 TRACE root Traceback (most recent call last):
2015-11-08 12:57:27.893 786 TRACE root File "/usr/bin/keystone-all", line 95, in serve
2015-11-08 12:57:27.893 786 TRACE root server.launch_with(launcher)
2015-11-08 12:57:27.893 786 TRACE root File "/usr/bin/keystone-all", line 67, in launch_with
2015-11-08 12:57:27.893 786 TRACE root self.server.listen()
2015-11-08 12:57:27.893 786 TRACE root File "/usr/lib/python2.6/site-packages/keystone/common/environment/eventlet_server.py", line 98, in listen
2015-11-08 12:57:27.893 786 TRACE root backlog=backlog)
2015-11-08 12:57:27.893 786 TRACE root File "/usr/lib/python2.6/site-packages/eventlet/convenience.py", line 39, in listen
2015-11-08 12:57:27.893 786 TRACE root sock.bind(addr)
2015-11-08 12:57:27.893 786 TRACE root File "<string>", line 1, in bind
2015-11-08 12:57:27.893 786 TRACE root error: [Errno 98] Address already in use
2015-11-08 12:57:27.893 786 TRACE root
2015-11-08 12:57:27.895 786 CRITICAL keystone [-] error: [Errno 98] Address already in use
2015-11-08 12:57:27.895 786 TRACE keystone Traceback (most recent call last):
2015-11-08 12:57:27.895 786 TRACE keystone File "/usr/bin/keystone-all", line 168, in <module>
2015-11-08 12:57:27.895 786 TRACE keystone serve(*servers)
2015-11-08 12:57:27.895 786 TRACE keystone File "/usr/bin/keystone-all", line 95, in serve
2015-11-08 12:57:27.895 786 TRACE keystone server.launch_with(launcher)
2015-11-08 12:57:27.895 786 TRACE keystone File "/usr/bin/keystone-all", line 67, in launch_with
2015-11-08 12:57:27.895 786 TRACE keystone self.server.listen()
2015-11-08 12:57:27.895 786 TRACE keystone File "/usr/lib/python2.6/site-packages/keystone/common/environment/eventlet_server.py", line 98, in listen
2015-11-08 12:57:27.895 786 TRACE keystone backlog=backlog)
2015-11-08 12:57:27.895 786 TRACE keystone File "/usr/lib/python2.6/site-packages/eventlet/convenience.py", line 39, in listen
2015-11-08 12:57:27.895 786 TRACE keystone sock.bind(addr)
2015-11-08 12:57:27.895 786 TRACE keystone File "<string>", line 1, in bind
2015-11-08 12:57:27.895 786 TRACE keystone error: [Errno 98] Address already in use
2015-11-08 12:57:27.895 786 TRACE keystone

But seems that 35357 port is free:
[root@nailgun ~]# netstat -nap | grep 35357
[root@nailgun ~]#

[Andrey Sledzinskiy (asledzinskiy)]

Set to high because it reproduced only on one thread

[Bogdan Dobrelya (bogdando)]

We should use the class https://github.com/openstack/fuel-library/blob/master/deployment/puppet/openstack/manifests/reserved_ports.pp in order to configure ephemeral ports assignments

[Matthew Mosesohn (raytrac3r)]

It might not be so simple, Bogdan. Docker attaches itself to exposed ports from containers (even in host networking) to prevent container start before securing a reserved port. We can try and see how it works, though.

[Matthew Mosesohn (raytrac3r)]

FYI please don't run bootstrap_admin_node.sh. It will reset all settings to defaults, possibly breaking network config.
I'll research the keystone issue itself, though.

[Matthew Mosesohn (raytrac3r)]

I tried the steps suggested in the description, but I could not reproduce this bug. Keystone deployed just fine.

[Andrey Sledzinskiy (asledzinskiy)]

This issue also didn't show up again on 7.0 SWARM with MU but it doesn't mean there is no issue