Keystone's IANA-assigned default port in linux local ephemeral port range: [Errno 98] Address already in use

[Clark Boylan]

The linux ip port local range is 32768 to 61000 as reported by sysctl:

$ sysctl -a | grep ip_local_port_range
net.ipv4.ip_local_port_range = 32768 61000

Keystone's default port as assigned by IANA is 35357. It is therefore possible that keystone will fail to start because some application has a socket open on port 35357. We believe this is the case logged at http://logs.openstack.org/09/57509/2/gate/gate-tempest-devstack-vm-large-ops/1171354/logs/screen-key.txt.gz?level=TRACE.

To fix this devstack should shift the ephemeral port range to 49152 to 61000 to avoid IANA port allocations and to avoid linux private port ranges.

Additionally keystone should document this fact so that deployers are aware of this and know to work around the funny linux default range.

[Dolph Mathews (dolph)]

Given that 35357 is keystone's IANA assigned port, I don't think it's reasonable to expect keystone to change ports in the short term. In the long term (after dropping API v2 support), keystone will be deployable on any one port. Using 35357 theoretically aids discoverability a tiny bit, but there are other means of discoverability that we can investigate in the future (such as RFC 2782).

[Dolph Mathews (dolph)]

Missed the documentation requirement in the last paragraph. 100% agree!

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/57577

[Dolph Mathews (dolph)]

In keystone and openstack-manuals, I think we should add documentation recommending that deployers explicitly exclude port 35357 from the available ephemeral ports, regardless of what the ephemeral port range (ip_local_port_range ) is configured to be. In a clean linux environment, this would be as simple as:

    echo 35357 > /proc/sys/net/ipv4/ip_local_reserved_ports

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/58013

[Anne Gentle (annegentle)]

This should go in the Configuration Reference, Operations Guide, and Cloud Admin Guide, preferably in a common location.

[Dolph Mathews (dolph)]

Ignore the details of comment #4 above -- a better solution is developing in the above code reviews.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/59528

[Matt Riedemann (mriedem)]

Doesn't show up often but it does show up:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwia2V5c3RvbmUtYWxsIHNlcnZlIEZhaWxlZCB0byBzdGFydCB0aGUgYWRtaW4gc2VydmVyXCIgQU5EIGZpbGVuYW1lOlwibG9ncy9zY3JlZW4ta2V5LnR4dFwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzg2NjE2NDQ4ODU5fQ==

[Matt Riedemann (mriedem)]

Pushed a patch to track this in elastic-recheck: https://review.openstack.org/#/c/60917/

[Morgan Fainberg (mdrnstm)]

Matt, Thanks for adding that!

[Sean Dague (sdague)]

so we just got a failure spike on this, anyone know why?

[Clark Boylan (cboylan)]

The spike may be related to jenkins01 crashing. If it reused any slave nodes when it came back up (because nodepool didn't get the appropriate events due to jenkins crashing) then keystone could have been running on those nodes preventing it from starting when the second jobs ran.

[Sean Dague (sdague)]

Is there a ci bug for the jenkins crash? Because right now we are e-r assigning the jenkins crash and slave reuse to this bug, which is really not the right thing to do.

[Tom Fifield (fifieldt)]

In openstack-manuals, this will now be dealt holistically across all services by this bug : https://bugs.launchpad.net/openstack-manuals/+bug/1261617 " Provide information and instructions on default ports and firewall rules "

[Joe Gordon (jogo)]

We are seeing this in the gate again

[Joe Gordon (jogo)]

Adam young suggested setting KEYSTONE_AUTH_PORT=32635 in gate

[Adam Young (ayoung)]

Any non-claimed port will work: look in /etc/services

We want to make sure it is not a port used by openstack. So that rules out the ports for LDAP, MySQL, PostGresql, Memcached, and then any of the openstack services. These should all be registered in /etc/services.

There are many unclaimed ports under 32768. What is essential is that the machines consider the port non-ephemeral, and it not conflict.

 looks like there is an open range from
iceedcp_rx 31949
to
iracinghelper 32034 so

how about 32999.

This is just for the gate, not a change we are going to make to devstack proper or Keystone

[Dolph Mathews (dolph)]

This bug has a very consistent rate of occurrence, but the elastic-recheck query for this bug was a bit too vague, and picked up several false positives. Eliminating the false positives in the query reduced the 7 day impact of this bug from 15 hits to just 6: https://review.openstack.org/69997

[Joe Gordon (jogo)]

Note: logs from keystone meeting http://eavesdrop.openstack.org/meetings/keystone/2014/keystone.2014-01-21-18.01.log.html

[Dolph Mathews (dolph)]

Keystone docs should be revised to include the caveats around usage of port 35357 in production, along with some pointers to suggested remedies.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/57577
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=6c57fbab26e40af5c5b19b46fb3da39341f34dab
Submitter: Jenkins
Branch: master

commit 6c57fbab26e40af5c5b19b46fb3da39341f34dab
Author: Clark Boylan <email address hidden>
Date: Wed Nov 20 17:00:21 2013 -0800

    Set keystone admin_bind_host to KEYSTONE_SERVICE_HOST

    On Linux ports 32768-61000 can be used by just about
    anything needing a socket. Keystone's IANA assigned port is 35357.
    Occasionally something else will be using port 35357 first because Linux
    allows this. Workaround is to bind to port 127.0.0.1 instead of 0.0.0.0.
    $KEYSTONE_SERVICE_HOST gets its value from $SERVICE_HOST which is set to
    127.0.0.1 in the gate.

    "Ephemeral (client) ports will *never* be sourced from 0.0.0.0, and are
    uniquely identified by the full connection five-tuple (proto, src IP,
    src port, dst IP, dst port) anyway, allowing them to overlap src IP/src
    port as long as proto/dst IP/dst port are different. Thus it is up to
    keystone/devstack to bind more appropriately and not use wildcard bind
    addresses unless explicitly necessary for some reason. For example, in
    the log output, the URLs are configured with dst IPs of 127.0.0.1
    anyway, so binding explicitly to localhost would change nothing, while
    skirting this particular edge case nicely." ~Evan Callicoat

    This doesn't fix bug 1253482 it works around it while a better solution
    is prepared (running keystone behind apache in devstack).

    Co-Authored-By: Joe Gordon <email address hidden>
    Change-Id: I112309661dadf8b753c3311182f82464d9d3595e
    Related-bug: #1253482

[Dolph Mathews (dolph)]

The spike in today's elastic rechecks against this bug are actually instances of bug 1277507, and a result of the elastic recheck query as described in https://bugs.launchpad.net/devstack/+bug/1253482/comments/19

[Sean Dague (sdague)]

This remains a gate reset issue. So I'm making this critical for keystone. There needs to be some better retry logic around this or some where to make this this not continue to be a race in the gate.

[Jeremy Stanley (fungi)]

It definitely seems to still not be fixed in stable/havana either, since it's breaking grenade jobs during the setup phase:

(keystone.token.provider): 2014-03-14 12:08:46,033 WARNING provider get_token_provider keystone.conf [signing] token_format is deprecated in favor of keystone.conf [token] provider
(keystone.common.environment.eventlet_server): 2014-03-14 12:08:46,128 INFO eventlet_server start Starting /opt/stack/old/keystone/bin/keystone-all on 0.0.0.0:35357
(keystone): 2014-03-14 12:08:46,128 CRITICAL log logging_excepthook [Errno 98] Address already in use
key failed to start

http://logs.openstack.org/76/80476/1/gate/gate-grenade-dsvm/ae07834/logs/old/screen-key.txt.gz?level=INFO

[Matt Riedemann (mriedem)]

Looks like our e-r query isn't working in all cases anymore:

http://logs.openstack.org/15/84215/4/gate/gate-tempest-dsvm-neutron/e82ae58/logs/screen-key.txt.gz

At least I didn't get any message from e-r in this patch: https://review.openstack.org/#/c/84215

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/havana
Review: https://review.openstack.org/93641

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/93641
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=e4dda729ef9670927978f49c9d21f160c0457efb
Submitter: Jenkins
Branch: stable/havana

commit e4dda729ef9670927978f49c9d21f160c0457efb
Author: Clark Boylan <email address hidden>
Date: Wed Nov 20 17:00:21 2013 -0800

    Set keystone admin_bind_host to KEYSTONE_SERVICE_HOST

    NOTE: squashed with follow-up Make admin_bind_host configurable
    to avoid regression bug #1283803

    On Linux ports 32768-61000 can be used by just about
    anything needing a socket. Keystone's IANA assigned port is 35357.
    Occasionally something else will be using port 35357 first because Linux
    allows this. Workaround is to bind to port 127.0.0.1 instead of 0.0.0.0.
    $KEYSTONE_SERVICE_HOST gets its value from $SERVICE_HOST which is set to
    127.0.0.1 in the gate.

    "Ephemeral (client) ports will *never* be sourced from 0.0.0.0, and are
    uniquely identified by the full connection five-tuple (proto, src IP,
    src port, dst IP, dst port) anyway, allowing them to overlap src IP/src
    port as long as proto/dst IP/dst port are different. Thus it is up to
    keystone/devstack to bind more appropriately and not use wildcard bind
    addresses unless explicitly necessary for some reason. For example, in
    the log output, the URLs are configured with dst IPs of 127.0.0.1
    anyway, so binding explicitly to localhost would change nothing, while
    skirting this particular edge case nicely." ~Evan Callicoat

    This doesn't fix bug 1253482 it works around it while a better solution
    is prepared (running keystone behind apache in devstack).

    Co-Authored-By: Joe Gordon <email address hidden>
    Change-Id: I112309661dadf8b753c3311182f82464d9d3595e
    Related-bug: #1253482
    (cherry picked from commit 6c57fbab26e40af5c5b19b46fb3da39341f34dab)
    (cherry picked from commit 041fa712472d887550a540dd50ade546f847c6b4)

[Vish Ishaya (vishvananda)]

The keystone port can be reserved quite easily in linux via:

echo "35357" | sudo cat /proc/sys/net/ipv4/ip_local_reserved_ports

[Morgan Fainberg (mdrnstm)]

@Vish,

That has been commented as a "fix" but was disliked for a number of reasons. I agree it is the easiest fix. (Unfortunately, IANA and linux are at odds here and we're losing because of it).

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/58013
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=1b338ab0bf8cbaef03f59d7d3a474ccbed2c9029
Submitter: Jenkins
Branch: master

commit 1b338ab0bf8cbaef03f59d7d3a474ccbed2c9029
Author: Dolph Mathews <email address hidden>
Date: Fri Nov 22 13:38:47 2013 -0600

    recommend excluding 35357 from ephemeral ports

    Change-Id: I56c3b526ac4af255dd31ce0bd0e0f2ce128cb6fe
    Closes-Bug: 1253482

[Matt Riedemann (mriedem)]

The e-r query on this bug seems to not be working anymore, it shows no hits in 14 days but I hit it here:

http://logs.openstack.org/93/98493/1/check/check-swift-dsvm-functional/ffb861b/logs/screen-key.txt.gz#_2014-06-09_14_08_14_587

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/99779

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/99779
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=6cae83efd72133adae891da0665f51f526705592
Submitter: Jenkins
Branch: master

commit 6cae83efd72133adae891da0665f51f526705592
Author: Morgan Fainberg <email address hidden>
Date: Thu Jun 12 15:08:48 2014 -0700

    Reserve Keystone ports from the ephemeral range

    Reserve Keystone ports from the ephemeral range as early as reasonably
    possible in the fixup_stuff.sh process to reduce the likelihood that the
    port will be in use. This does not completely resolve the issue
    where Keystone's IANA assigned port falls into Linux's ephemeral
    range, but this should reduce the occurrences. The default ports
    are 35357 and 35358.

    Change-Id: I8cfb53d8f90c1ff1fb1083c59fefabca3d14323b
    Partial-Bug: #1253482

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/130187

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/130187
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=80cad72055db14e6bc7c610c470f9977266e1c65
Submitter: Jenkins
Branch: stable/icehouse

commit 80cad72055db14e6bc7c610c470f9977266e1c65
Author: Morgan Fainberg <email address hidden>
Date: Thu Jun 12 15:08:48 2014 -0700

    Reserve Keystone ports from the ephemeral range

    Reserve Keystone ports from the ephemeral range as early as reasonably
    possible in the fixup_stuff.sh process to reduce the likelihood that the
    port will be in use. This does not completely resolve the issue
    where Keystone's IANA assigned port falls into Linux's ephemeral
    range, but this should reduce the occurrences. The default ports
    are 35357 and 35358.

    Change-Id: I8cfb53d8f90c1ff1fb1083c59fefabca3d14323b
    Partial-Bug: #1253482
    (cherry picked from commit 6cae83efd72133adae891da0665f51f526705592)