Unable to get version from keystone URL

[Erich Cordoba]

Brief Description
-----------------
Openstack commands fails with this error:

1) Failed to discover available identity versions when contacting http://keystone.openstack.svc.cluster.local/v3. Attempting to parse version from URL.
Unable to establish connection to http://keystone.openstack.svc.cluster.local/v3/auth/tokens: HTTPConnectionPool(host='keystone.openstack.svc.cluster.local', port=80): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9a819b0250>: Failed to establish a new connection: [Errno -2] Name or service not known',))

Severity
--------
Major

Steps to Reproduce
------------------
Run the Sanity-Openstack test subset.

python runner.py --run-suite Sanity-Test.Sanity-Openstack

Expected Behavior
------------------
The Sanity tests should pass.

Actual Behavior
----------------
Several openstack commands fails with the following error:

1) Failed to discover available identity versions when contacting http://keystone.openstack.svc.cluster.local/v3. Attempting to parse version from URL.
Unable to establish connection to http://keystone.openstack.svc.cluster.local/v3/auth/tokens: HTTPConnectionPool(host='keystone.openstack.svc.cluster.local', port=80): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9a819b0250>: Failed to establish a new connection: [Errno -2] Name or service not known',))

2) Failed to discover available identity versions when contacting http://keystone.openstack.svc.cluster.local/v3. Attempting to parse version from URL.
Unable to establish connection to http://keystone.openstack.svc.cluster.local/v3/auth/tokens: HTTPConnectionPool(host='keystone.openstack.svc.cluster.local', port=80): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd16dd82e90>: Failed to establish a new connection: [Errno -2] Name or service not known',))

3) Failed to discover available identity versions when contacting http://keystone.openstack.svc.cluster.local/v3. Attempting to parse version from URL.
    [ Message content over the limit has been removed. ]
.../v3/auth/tokens: HTTPConnectionPool(host='keystone.openstack.svc.cluster.local', port=80): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1fab00fd10>: Failed to establish a new connection: [Errno -2] Name or service not known',))

7) Failed to discover available identity versions when contacting http://keystone.openstack.svc.cluster.local/v3. Attempting to parse version from URL.
Unable to establish connection to http://keystone.openstack.svc.cluster.local/v3/auth/tokens: HTTPConnectionPool(host='keystone.openstack.svc.cluster.local', port=80): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f589757e150>: Failed to establish a new connection: [Errno -2] Name or service not known',))

8) Failed to discover available identity versions when contacting http://keystone.openstack.svc.cluster.local/v3. Attempting to parse version from URL.
Unable to establish connection to http://keystone.openstack.svc.cluster.local/v3/auth/tokens: HTTPConnectionPool(host='keystone.openstack.svc.cluster.local', port=80): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7ff67f6e10>: Failed to establish a new connection: [Errno -2] Name or service not known',))

Reproducibility
---------------

System Configuration
--------------------
Virtual Dedicated Storage

Branch/Pull Time/Commit
-----------------------
OS="centos"
SW_VERSION="19.01"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20190604T144018Z"

JOB="STX_build_master_master"
<email address hidden>"
BUILD_NUMBER="128"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2019-06-04 14:40:18 +0000"

Timestamp/Logs
--------------
Working on get more logs.

Test Activity
-------------
[Sanity]

[Ghada Khalil (gkhalil)]

What was the status of the pods and SM services when the openstack cmds were failing?
What were the cmds executed by sanity leading up to this failure?

FYI, there are already other bug reports with a similar failure signature. Without knowing what the status of the system is at the time of the failure, it's not possible to debug

[Ghada Khalil (gkhalil)]

Collect logs are also needed to investigate further

[Ghada Khalil (gkhalil)]

Any update on this issue?

[Erich Cordoba (ericho)]

This issue was affecting Sanity tests, this hasn't been present in the last two week executions. losing as it cannot be reproduced.

[Erich Cordoba (ericho)]

Issue not reproducible In automation.