undercloud deploy fails with keystone token issue error

[Pradeep Kilambi]

When deploying undercloud with oooq, frequently run into failures die to keystone token issue

2017-06-23 17:53:35 | 2017-06-23 17:53:35,790 INFO: ESC[mNotice: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[a
dmin]/description: description changed 'Bootstrap project for initializing the cloud.' to 'admin tenant'ESC[0m
2017-06-23 17:56:39 | 2017-06-23 17:56:39,092 INFO: ESC[1;31mError: /Stage[main]/Keystone::Roles::Admin/Keystone_user[admin]: Could not evaluate: Execution of '/bin/openstack token issue --format value' returned 1: Unable to establish connection to https://192.168.24.2:13000/v3/auth/tokens: HTTPSConnectionPool(host='192.168.24.2', port=13000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x396fad0>: Failed to establish a new connection: [Errno 111] Connection refused',)) (tried 30, for a total of 170 seconds)ESC[0m

[Alex Schultz (alex-schultz)]

So i took a look at the logs, and I don't see apache actually listening on port 13000. It seems that the undercloud is listening on 5000 instead.

It seems that https://github.com/openstack/instack-undercloud/blob/master/elements/puppet-stack-config/puppet-stack-config.pp#L328-L334 is being configured but ssl is not actually enabled.

[Alex Schultz (alex-schultz)]

So ssl is supposed to be handled by haproxy, but i don't see haproxy listening on 13000 so something appears to be up with the haproxy. There's a bunch of defunct instack-haproxy processes in the process list. Still investigating but it looks to be related to haproxy

[Alex Schultz (alex-schultz)]

Wonder if it's related to these AVCs. The upstream CI doesn't run with selinux enabled which may be why we don't see it:

./auditd/ausearch_--input-logs_-m_avc_user_avc_-ts_today:type=USER_AVC msg=audit(1498240010.178:1979): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='avc: denied { reload } for auid=n/a uid=0 gid=0 path="/usr/lib/systemd/system/haproxy.service" cmdline="systemctl reload haproxy" scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:haproxy_unit_file_t:s0 tclass=service exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?'
./auditd/ausearch_--input-logs_-m_avc_user_avc_-ts_today:type=USER_AVC msg=audit(1498240011.424:1980): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='avc: denied { status } for auid=n/a uid=0 gid=0 path="/usr/lib/systemd/system/haproxy.service" cmdline="systemctl reload haproxy" scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:haproxy_unit_file_t:s0 tclass=service exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?'

[Alex Schultz (alex-schultz)]

Closing this as the actual error was an selinux problem and the reported error message generally appears if something else goes wrong during the deployment.