Existing Rally Deployment complains about wrong credentials although openstackclient is operational

[Francisco J. Lopez]

Check Rally Database to get Auth issue cleared

Authentication Issue:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

[root@cci06-rally01 database]# rally deployment check
2017-06-06 10:16:54.276 32584 RALLYDEBUG rally.api [-] INFO logs from urllib3 and requests module are hide.
2017-06-06 10:16:54.277 32584 RALLYDEBUG rally.api [-] urllib3 insecure warnings are hidden.
2017-06-06 10:16:54.277 32584 RALLYDEBUG rally.api [-] ERROR log from boto module is hide.
2017-06-06 10:16:54.277 32584 DEBUG oslo_db.api [-] Loading backend 'sqlalchemy' from 'rally.common.db.sqlalchemy.api' _load_backend /root/rally/rally-master/venv/cci06/lib/python3.5/site-packages/oslo_db/
--------------------------------------------------------------------------------
Platform openstack:
--------------------------------------------------------------------------------

Error while checking admin credentials:
Traceback (most recent call last):
  File "/root/rally/rally-master/venv/cci06/lib/python3.5/site-packages/rally/osclients.py", line 803, in verified_keystone
    self.keystone.auth_ref.role_names]:
  File "/root/rally/rally-master/venv/cci06/lib/python3.5/site-packages/rally/osclients.py", line 236, in auth_ref
    self.cache["keystone_auth_ref"] = plugin.get_access(sess)
  File "/root/rally/rally-master/venv/cci06/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 136, in get_access
    self.auth_ref = self.get_auth_ref(session)
  File "/root/rally/rally-master/venv/cci06/lib/python3.5/site-packages/keystoneauth1/identity/generic/base.py", line 198, in get_auth_ref
    return self._plugin.get_auth_ref(session, **kwargs)
  File "/root/rally/rally-master/venv/cci06/lib/python3.5/site-packages/keystoneauth1/identity/v3/base.py", line 167, in get_auth_ref
    authenticated=False, log=False, **rkwargs)
  File "/root/rally/rally-master/venv/cci06/lib/python3.5/site-packages/keystoneauth1/session.py", line 766, in post
    return self.request(url, 'POST', **kwargs)
  File "/root/rally/rally-master/venv/cci06/lib/python3.5/site-packages/positional/__init__.py", line 101, in inner
    return wrapped(*args, **kwargs)
  File "/root/rally/rally-master/venv/cci06/lib/python3.5/site-packages/keystoneauth1/session.py", line 655, in request
    raise exceptions.from_response(resp, method, url)
keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-e54d993e-3e8e-446a-a0bd-05d407d93c6c)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/rally/rally-master/venv/cci06/lib/python3.5/site-packages/rally/api.py", line 192, in check
    credential["admin"].verify_connection()
  File "/root/rally/rally-master/venv/cci06/lib/python3.5/site-packages/rally/plugins/openstack/credential.py", line 89, in verify_connection
    self.clients().verified_keystone()
  File "/root/rally/rally-master/venv/cci06/lib/python3.5/site-packages/rally/osclients.py", line 807, in verified_keystone
    raise exceptions.InvalidEndpointsException()
rally.exceptions.InvalidEndpointsException: wrong keystone credentials specified in your endpoint properties. (HTTP 401)

[root@cci06-rally01 database]#

#################################################################################

[root@cci06-rally01 database]# rally deployment list
2017-06-06 10:17:43.318 32589 RALLYDEBUG rally.api [-] INFO logs from urllib3 and requests module are hide.
2017-06-06 10:17:43.318 32589 RALLYDEBUG rally.api [-] urllib3 insecure warnings are hidden.
2017-06-06 10:17:43.318 32589 RALLYDEBUG rally.api [-] ERROR log from boto module is hide.
2017-06-06 10:17:43.319 32589 DEBUG oslo_db.api [-] Loading backend 'sqlalchemy' from 'rally.common.db.sqlalchemy.api' _load_backend /root/rally/rally-master/venv/cci06/lib/python3.5/site-packages/oslo_db/api.py:237
+--------------------------------------+----------------------------+----------------+------------------+--------+
| uuid | created_at | name | status | active |
+--------------------------------------+----------------------------+----------------+------------------+--------+
| fc8cb090-0384-4e0b-ae1f-9961392b4979 | 2017-06-06 06:39:56.131878 | CCI01-LinuxEng | deploy->finished | |
| 6f725f19-d0a9-4d1b-9919-1632ff988f18 | 2017-06-06 06:40:27.726704 | CCI06-LinuxEng | deploy->finished | * |
+--------------------------------------+----------------------------+----------------+------------------+--------+

##################################################################################

[root@cci06-rally01 database]# rally deployment show CCI06-LinuxEng
2017-06-06 10:18:27.947 32600 RALLYDEBUG rally.api [-] INFO logs from urllib3 and requests module are hide.
2017-06-06 10:18:27.947 32600 RALLYDEBUG rally.api [-] urllib3 insecure warnings are hidden.
2017-06-06 10:18:27.947 32600 RALLYDEBUG rally.api [-] ERROR log from boto module is hide.
2017-06-06 10:18:27.948 32600 DEBUG oslo_db.api [-] Loading backend 'sqlalchemy' from 'rally.common.db.sqlalchemy.api' _load_backend /root/rally/rally-master/venv/cci06/lib/python3.5/site-packages/oslo_db/api.py:237
+--------------------------------------------------+----------+----------+-------------+-------------+---------------+
| auth_url | username | password | tenant_name | region_name | endpoint_type |
+--------------------------------------------------+----------+----------+-------------+-------------+---------------+
| https://cci06-ostctrl-oc.cloud.internal:35357/v3 | admin | *** | LinuxEng | CCI06 | public |
+--------------------------------------------------+----------+----------+-------------+-------------+---------------+

##################################################################################

[root@cci06-rally01 database]# rally deployment config CCI06-LinuxEng
2017-06-06 10:18:44.949 32610 RALLYDEBUG rally.api [-] INFO logs from urllib3 and requests module are hide.
2017-06-06 10:18:44.949 32610 RALLYDEBUG rally.api [-] urllib3 insecure warnings are hidden.
2017-06-06 10:18:44.949 32610 RALLYDEBUG rally.api [-] ERROR log from boto module is hide.
2017-06-06 10:18:44.950 32610 DEBUG oslo_db.api [-] Loading backend 'sqlalchemy' from 'rally.common.db.sqlalchemy.api' _load_backend /root/rally/rally-master/venv/cci06/lib/python3.5/site-packages/oslo_db/api.py:237
{
    "creds": {
        "openstack": {
            "admin": {
                "password": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
                "project_domain_name": "Linux1Eng",
                "project_name": "Linux1Eng",
                "user_domain_name": "default",
                "username": "admin"
            },
            "auth_url": "https://cci06-oc.cloud.internal:35357/v3",
            "endpoint_type": "public",
            "https_cacert": "/etc/pki/CA/certs/CCI06_Root_CA.pem",
            "https_insecure": false,
            "region_name": "CCI06"
        }
    },
    "type": "ExistingCloud"
}

###################################################################################

-rw-r--r--. 1 root root 57344 Jun 6 08:40 rally.sqlite
[root@cci06-rally01 database]# sqlite3 rally.sqlite
SQLite version 3.7.17 2013-05-20 00:56:22
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> .databases
seq name file
--- --------------- ----------------------------------------------------------
0 main /root/rally/rally-master/venv/cci06/database/rally.sqlite
sqlite> .explain ON
sqlite> .header on
sqlite> .tables
alembic_version subtasks verifications workloaddata
deployments tags verifiers workloads
resources tasks workers
sqlite> select * from resources
   ...> ;
sqlite> select * from deployments;
crea updated_at id uuid pare name st completed_at config s
---- ------------- ---- ---- ---- ------------- -- ------------- --------------------------------------------------------------------------------------------- -
2017-06-06 06:39:56.131878 2017-06-06 06:39:56.535932 1 fc8cb090-0384-4e0b-ae1f-9961392b4979 CCI01-LinuxEng 2017-06-06 08:39:56.282848 2017-06-06 08:39:56.418755 {"type": "ExistingCloud", "creds": {"openstack": {"auth_ur
2017-06-06 06:40:27.726704 2017-06-06 06:40:28.085396 2 6f725f19-d0a9-4d1b-9919-1632ff988f18 CCI06-LinuxEng 2017-06-06 08:40:27.864816 2017-06-06 08:40:27.978465 {"type": "ExistingCloud", "creds": {"openstack": {"auth_ur
sqlite> select config from deployments;
conf
----
{"type": "ExistingCloud", "creds": {"openstack": {"auth_url": "http://openstack.internal.com:35357/v2.0/", "region_name": "RegionOne", "endpoint_type": "public", "admin": {"username": "admin", "password": "}
{"type": "ExistingCloud", "creds": {"openstack": {"auth_url": "https://cci06-ostctrl-oc.cloud.internal:35357/v3", "region_name": "CCI06", "endpoint_type": "public", "admin": {"username": "admin", "password": "", "user_d}

[Francisco J. Lopez (r-openstack)]

[root@cci06-rally01 database(openstack_admin_LinuxEng)]# rally --debug deployment check
2017-06-07 13:51:41.914 6587 RALLYDEBUG rally.api [-] INFO logs from urllib3 and requests module are hide.
2017-06-07 13:51:41.914 6587 RALLYDEBUG rally.api [-] urllib3 insecure warnings are hidden.
2017-06-07 13:51:41.914 6587 RALLYDEBUG rally.api [-] ERROR log from boto module is hide.
2017-06-07 13:51:41.915 6587 DEBUG oslo_db.api [-] Loading backend 'sqlalchemy' from 'rally.common.db.sqlalchemy.api' _load_backend /root/rally/rally-master/venv/cci06/lib/python3.5/site-packages/oslo_db/api.py:237
/root/rally/rally-master/venv/cci06/lib/python3.5/site-packages/requests/packages/urllib3/connection.py:344: SubjectAltNameWarning: Certificate for cci06-ostctrl-oc.cloud.internal has no `subjectAltName`, falling back to check for a `co)
  SubjectAltNameWarning
/root/rally/rally-master/venv/cci06/lib/python3.5/site-packages/requests/packages/urllib3/connection.py:344: SubjectAltNameWarning: Certificate for cci06-ostctrl-oc.cloud.internal has no `subjectAltName`, falling back to check for a `co)
  SubjectAltNameWarning
/root/rally/rally-master/venv/cci06/lib/python3.5/site-packages/requests/packages/urllib3/connection.py:344: SubjectAltNameWarning: Certificate for cci06-ostctrl-oc.cloud.internal has no `subjectAltName`, falling back to check for a `co)
  SubjectAltNameWarning
--------------------------------------------------------------------------------
Platform openstack:
--------------------------------------------------------------------------------

Error while checking admin credentials:
Traceback (most recent call last):
  File "/root/rally/rally-master/venv/cci06/lib/python3.5/site-packages/rally/osclients.py", line 803, in verified_keystone
    self.keystone.auth_ref.role_names]:
  File "/root/rally/rally-master/venv/cci06/lib/python3.5/site-packages/rally/osclients.py", line 236, in auth_ref
    self.cache["keystone_auth_ref"] = plugin.get_access(sess)
  File "/root/rally/rally-master/venv/cci06/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 136, in get_access
    self.auth_ref = self.get_auth_ref(session)
  File "/root/rally/rally-master/venv/cci06/lib/python3.5/site-packages/keystoneauth1/identity/generic/base.py", line 198, in get_auth_ref
    return self._plugin.get_auth_ref(session, **kwargs)
  File "/root/rally/rally-master/venv/cci06/lib/python3.5/site-packages/keystoneauth1/identity/v3/base.py", line 167, in get_auth_ref
    authenticated=False, log=False, **rkwargs)
  File "/root/rally/rally-master/venv/cci06/lib/python3.5/site-packages/keystoneauth1/session.py", line 766, in post
    return self.request(url, 'POST', **kwargs)
  File "/root/rally/rally-master/venv/cci06/lib/python3.5/site-packages/positional/__init__.py", line 101, in inner
    return wrapped(*args, **kwargs)
  File "/root/rally/rally-master/venv/cci06/lib/python3.5/site-packages/keystoneauth1/session.py", line 655, in request
    raise exceptions.from_response(resp, method, url)
keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-47cf0b64-4501-4eb8-8865-78b766abb768)

During handling of ...