Deployment of simple cluster failed with Could not start Service[keystone]: Execution of '/sbin/service openstack-keystone start' returned 1:

[Andrey Sledzinskiy]

{

    "build_id": "2014-08-03_02-01-14",
    "ostf_sha": "a3fa823ea0e4e03beb637ae07a91adea82c33182",
    "build_number": "389",
    "auth_required": true,
    "api": "1.0",
    "nailgun_sha": "bd0127be0061029f9f910547db5e633c82244942",
    "production": "docker",
    "fuelmain_sha": "e99879292cf6e96b8991300d947df76b69134bb1",
    "astute_sha": "ce86172e77661026c91fdf1ff8066d7df1f7d89d",
    "feature_groups": [
        "mirantis"
    ],
    "release": "5.1",
    "fuellib_sha": "4e3fdd75f8dabde8e5d07067545d8043a70a176b"

}

Steps:
1. Create next cluster - Centos, Simple, Neutron GRE, 1 controller, 1 compute
2. Run deployment

Expected - deployment passed
Actual - deploymetn failed with error in puppet log on controller-1 (node-3):
Could not start Service[keystone]: Execution of '/sbin/service openstack-keystone start' returned 1:

Error in keystone-all log:
2014-08-03 20:19:26 ERROR

root [-] Failed to start the admin server
2014-08-03 19:19:26.040 19128 TRACE root Traceback (most recent call last):
2014-08-03 19:19:26.040 19128 TRACE root File "/usr/bin/keystone-all", line 78, in serve
2014-08-03 19:19:26.040 19128 TRACE root server.start()
2014-08-03 19:19:26.040 19128 TRACE root File "/usr/lib/python2.6/site-packages/keystone/common/environment/eventlet_server.py", line 66, in start
2014-08-03 19:19:26.040 19128 TRACE root backlog=backlog)
2014-08-03 19:19:26.040 19128 TRACE root File "/usr/lib/python2.6/site-packages/eventlet/convenience.py", line 38, in listen
2014-08-03 19:19:26.040 19128 TRACE root sock.bind(addr)
2014-08-03 19:19:26.040 19128 TRACE root File "<string>", line 1, in bind
2014-08-03 19:19:26.040 19128 TRACE root error: [Errno 98] Address already in use
2014-08-03 19:19:26.040 19128 TRACE root

[Ryan Moe (rmoe)]

It looks like this is caused by RabbitMQ using port 35357 for its connection to epmd. The default local port range on linux is 32768-61000. In simple mode keystone will attempt to bind to 0.0.0.0.

root@node-8:~# cat /proc/sys/net/ipv4/ip_local_port_range
32768 61000

From the netstat output form the diagnostic snapshot:
tcp 0 0 127.0.0.1:35357 127.0.0.1:4369 ESTABLISHED 18490/beam

[Ryan Moe (rmoe)]

This port range can be controlled by the inet_dist_listen_min/max settings in rabbit.config

[Bogdan Dobrelya (bogdando)]

We do configure this port range as 41055 port in fuel manifests, see https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/rabbitmq/manifests/server.pp#L51-52
The root cause of issue should be the manual changing these values.
In a standard case there woun't be any collisions:
[root@node-2 ~]# ss -untap | grep 35357
tcp LISTEN 0 128 10.108.2.2:35357 *:* users:(("haproxy",2744,12))
tcp LISTEN 0 128 10.108.1.2:35357 *:* users:(("haproxy",2744,11))
tcp LISTEN 0 128 10.108.2.4:35357 *:* users:(("keystone-all",1937,4))
tcp TIME-WAIT 0 0 10.108.2.4:33074 10.108.2.2:35357
[root@node-2 ~]# ss -untap | grep 41055
tcp LISTEN 0 128 *:41055 *:* users:(("beam",1638,7))
tcp ESTAB 0 0 10.108.2.4:49155 10.108.2.5:41055 users:(("beam",1638,29))
tcp ESTAB 0 0 10.108.2.4:41055 10.108.2.3:57642 users:(("beam",1638,9))

[Timur Nurlygayanov (tnurlygayanov)]

Need to find the root of this issue, Heat OSTF/system tests fail with this error.

[Bogdan Dobrelya (bogdando)]

Agree, but https://bugs.launchpad.net/fuel/+bug/1352309/comments/2 is not a root cause for this case

[Ryan Moe (rmoe)]

Bogdan is correct. I missed those config options when I was looking into this. I deployed an environment with ISO 389 and couldn't reproduce this.

Timur, can you point us to other failures caused by this?

[Sergii Golovatiuk (sgolovatiuk)]

I had the similar behavior. To fix this problem for sure we need to specify

sysctl -w 'sys.net.ipv4.ip_local_reserved_ports=XXXXX' for all service ports that are higher 32768. We have another bug regarding this

https://bugs.launchpad.net/fuel/+bug/1353363

[Sergii Golovatiuk (sgolovatiuk)]

I am changing this bug as Invalid as this case happens very seldom. Also it has been superseded by https://bugs.launchpad.net/fuel/+bug/1353363