Distributed Cloud: distributed-cloud-services do not come up

[Kristine Bujold]

Brief Description
-----------------

dcorch reports constants failure, when looking at sm we also see the services are not up.

dcorch.log
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service [req-6140fce2-d6a3-4549-8c62-0b189cabde31 - - - - -] Error starting thread.: ConnectFailure: Unable to establish connection to http://192.168.204.2:25000/v3/services?: HTTPConnectionPool(host='192.168.204.2', port=25000): Max retries exceeded with url: /v3/services (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x5da7210>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service Traceback (most recent call last):
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 708, in run_service
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service service.start()
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/dcorch/engine/service.py", line 98, in start
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service self.init_qm()
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/dcorch/engine/service.py", line 85, in init_qm
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service self.qm = QuotaManager()
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/dcorch/engine/quota_manager.py", line 73, in __init__
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service self.endpoints = endpoint_cache.EndpointCache()
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/dcorch/common/endpoint_cache.py", line 46, in __init__
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service self._update_endpoints()
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/dcorch/common/endpoint_cache.py", line 136, in _update_endpoints
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service endpoint_map = EndpointCache._get_endpoint_from_keystone(self)
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/dcorch/common/endpoint_cache.py", line 107, in _get_endpoint_from_keystone
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service for service in self.keystone_client.services.list():
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service return wrapped(*args, **kwargs)
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/keystoneclient/v3/services.py", line 97, in list
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service **kwargs)
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/keystoneclient/base.py", line 75, in func
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service return f(*args, **new_kwargs)
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/keystoneclient/base.py", line 390, in list
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service self.collection_key)
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/keystoneclient/base.py", line 125, in _list
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service resp, body = self.client.get(url, **kwargs)
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 288, in get
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service return self.request(url, 'GET', **kwargs)
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 447, in request
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 192, in request
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service return self.session.request(url, method, **kwargs)
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service return wrapped(*args, **kwargs)
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 703, in request
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service resp = send(**kwargs)
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 777, in _send_request
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service raise exceptions.ConnectFailure(msg)
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service ConnectFailure: Unable to establish connection to http://192.168.204.2:25000/v3/services?: HTTPConnectionPool(host='192.168.204.2', port=25000): Max retries exceeded with url: /v3/services (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x5da7210>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2018-09-05 16:12:21.304 15894 ERROR oslo_service.service
2018-09-05 16:12:21.308 15894 ERROR dcorch.engine.service [-] Failed to stop engine service: 'NoneType' object has no attribute 'stop'
2018-09-05 16:12:21.308 15894 INFO dcorch.engine.service [-] All threads were gone, terminating engine
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service [req-f3a7ec7c-575d-49f4-80ec-8d952c6e12d2 - - - - -] Error starting thread.: ConnectFailure: Unable to establish connection to http://192.168.204.2:25000/v3/services?: HTTPConnectionPool(host='192.168.204.2', port=25000): Max retries exceeded with url: /v3/services (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x5c1c210>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service Traceback (most recent call last):
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 708, in run_service
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service service.start()
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/dcorch/engine/service.py", line 98, in start
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service self.init_qm()
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/dcorch/engine/service.py", line 85, in init_qm
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service self.qm = QuotaManager()
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/dcorch/engine/quota_manager.py", line 73, in __init__
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service self.endpoints = endpoint_cache.EndpointCache()
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/dcorch/common/endpoint_cache.py", line 46, in __init__
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service self._update_endpoints()
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/dcorch/common/endpoint_cache.py", line 136, in _update_endpoints
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service endpoint_map = EndpointCache._get_endpoint_from_keystone(self)
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/dcorch/common/endpoint_cache.py", line 107, in _get_endpoint_from_keystone
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service for service in self.keystone_client.services.list():
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service return wrapped(*args, **kwargs)
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/keystoneclient/v3/services.py", line 97, in list
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service **kwargs)
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/keystoneclient/base.py", line 75, in func
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service return f(*args, **new_kwargs)
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/keystoneclient/base.py", line 390, in list
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service self.collection_key)
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/keystoneclient/base.py", line 125, in _list
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service resp, body = self.client.get(url, **kwargs)
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 288, in get
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service return self.request(url, 'GET', **kwargs)
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 447, in request
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 192, in request
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service return self.session.request(url, method, **kwargs)
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service return wrapped(*args, **kwargs)
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 703, in request
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service resp = send(**kwargs)
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 777, in _send_request
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service raise exceptions.ConnectFailure(msg)
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service ConnectFailure: Unable to establish connection to http://192.168.204.2:25000/v3/services?: HTTPConnectionPool(host='192.168.204.2', port=25000): Max retries exceeded with url: /v3/services (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x5c1c210>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2018-09-05 16:12:23.572 16364 ERROR oslo_service.service
2018-09-05 16:12:23.575 16364 ERROR dcorch.engine.service [-] Failed to stop engine service: 'NoneType' object has no attribute 'stop'
2018-09-05 16:12:23.576 16364 INFO dcorch.engine.service [-] All threads were gone, terminating engine

[wrsroot@controller-0 dcorch(keystone_admin)]$ sudo sm-dump
Password:

-Service_Groups------------------------------------------------------------------------
oam-services active active
controller-services active active
cloud-services active active
patching-services active active
directory-services active active
web-services active active
vim-services active active
distributed-cloud-services active go-active failed
---------------------------------------------------------------------------------------

-Services------------------------------------------------------------------------------
oam-ip enabled-active enabled-active
management-ip enabled-active enabled-active
drbd-pg enabled-active enabled-active
drbd-rabbit enabled-active enabled-active
drbd-cgcs enabled-active enabled-active
drbd-platform enabled-active enabled-active
pg-fs enabled-active enabled-active
rabbit-fs enabled-active enabled-active
nfs-mgmt enabled-active enabled-active
cgcs-fs enabled-active enabled-active
platform-fs enabled-active enabled-active
postgres enabled-active enabled-active
rabbit enabled-active enabled-active
cgcs-export-fs enabled-active enabled-active
platform-export-fs enabled-active enabled-active
cgcs-nfs-ip enabled-active enabled-active
platform-nfs-ip enabled-active enabled-active
sysinv-inv enabled-active enabled-active
sysinv-conductor enabled-active enabled-active
mtc-agent enabled-active enabled-active
hbs-agent enabled-active enabled-active
hw-mon enabled-active enabled-active
dnsmasq enabled-active enabled-active
fm-mgr enabled-active enabled-active
keystone enabled-active enabled-active
glance-registry enabled-active enabled-active
glance-api enabled-active enabled-active
neutron-server enabled-active enabled-active
nova-api enabled-active enabled-active
nova-scheduler enabled-active enabled-active
nova-conductor enabled-active enabled-active
nova-console-auth enabled-active enabled-active
nova-novnc enabled-active enabled-active
cinder-api enabled-active enabled-active
cinder-scheduler enabled-active enabled-active
cinder-volume enabled-active enabled-active
ceilometer-agent-notification enabled-active enabled-active
open-ldap enabled-active enabled-active
snmp enabled-active enabled-active
lighttpd enabled-active enabled-active
horizon enabled-active enabled-active
patch-alarm-manager enabled-active enabled-active
drbd-cinder enabled-active enabled-active
cinder-lvm enabled-active enabled-active
iscsi enabled-active enabled-active
cinder-ip enabled-active enabled-active
vim enabled-active enabled-active
vim-api enabled-active enabled-active
vim-webserver enabled-active enabled-active
guest-agent enabled-active enabled-active
nova-api-proxy enabled-active enabled-active
haproxy enabled-active enabled-active
pxeboot-ip enabled-active enabled-active
drbd-extension enabled-active enabled-active
extension-fs enabled-active enabled-active
extension-export-fs enabled-active enabled-active
nova-placement-api enabled-active enabled-active
dcorch-engine enabled-active enabled-active
dcmanager-manager enabled-active enabled-active
dcmanager-api enabled-active enabled-active
dcorch-snmp enabled-active enabled-active
dcorch-sysinv-api-proxy enabled-active enabled-active
dcorch-nova-api-proxy enabled-active enabled-active
dcorch-neutron-api-proxy enabled-active enabled-active
dcorch-cinder-api-proxy enabled-active enabled-active
drbd-patch-vault enabled-active enabled-active
patch-vault-fs enabled-active enabled-active
dcorch-patch-api-proxy enabled-active enabled-active
dcorch-identity-api-proxy enabled-active initial

Steps to Reproduce
------------------
Install a Distributed Cloud system, do not need to have added subclouds.

Expected Behavior
------------------
All services managed by sm should start

Actual Behavior
----------------
distributed-cloud-services is in a failed state

Reproducibility
---------------
Always

System Configuration
--------------------
2 System Controllers

Branch/Pull Time/Commit
-----------------------
master

Timestamp/Logs
--------------
NA

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/600455

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/600455
Committed: https://git.openstack.org/cgit/openstack/stx-upstream/commit/?id=3fc7af31b8e5998741fad6b9b97905ce9f4eaedf
Submitter: Zuul
Branch: master

commit 3fc7af31b8e5998741fad6b9b97905ce9f4eaedf
Author: Jack Ding <email address hidden>
Date: Thu Sep 6 10:22:55 2018 -0400

    Make dcorch-identity-api-proxy script executable

    The ocf script dcorch-identity-api-proxy was incorrectly set to 644, as
    a result distributed-cloud-services can not come up.
    Change permission of the script to 755 so that it is executable.

    Closes-Bug: 1790950
    Change-Id: I68e3067878fa6686dfd19db07018a42f05d92906
    Signed-off-by: Jack Ding <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: f/centos75
Review: https://review.openstack.org/602666

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/602666
Committed: https://git.openstack.org/cgit/openstack/stx-upstream/commit/?id=e3bdf281f11d821bf01270862c2bee955dec8bd0
Submitter: Zuul
Branch: f/centos75

commit 55eaec60182484b5c8f11cfef590a03290d487dc
Author: Eddie Ramirez <email address hidden>
Date: Mon Aug 27 11:31:21 2018 -0700

    Remove dc_admin from python-horizion

    dc_admin is now part of the starlingx-dashboard package. This change
    removes it from the python horizon package. Without this change the
    horizon package will fail to build.

    Change-Id: Ia1470ae0b110df3400bc69f15b6a8e6c52bf355e
    Signed-off-by: David Sullivan <email address hidden>
    Depends-On: https://review.openstack.org/599475
    Story: 2002891
    Task: 26220

commit 3fc7af31b8e5998741fad6b9b97905ce9f4eaedf
Author: Jack Ding <email address hidden>
Date: Thu Sep 6 10:22:55 2018 -0400

    Make dcorch-identity-api-proxy script executable

    The ocf script dcorch-identity-api-proxy was incorrectly set to 644, as
    a result distributed-cloud-services can not come up.
    Change permission of the script to 755 so that it is executable.

    Closes-Bug: 1790950
    Change-Id: I68e3067878fa6686dfd19db07018a42f05d92906
    Signed-off-by: Jack Ding <email address hidden>

commit a0be02a79a02838f793ebdf6a15dd8a23e5a7755
Author: slin14 <email address hidden>
Date: Wed Sep 5 06:49:39 2018 +0800

    remove non exist path to avoid error message when build srpm

    Story: 2003518
    Task: 26180

    It is to avoid error message like below:
    "
    ERROR: (1281): Bad path 'openstack/openstack-swift' in file '/localdisk/designer/slin14/starlingx/cgcs-root/stx/stx-upstream/centos_pkg_dirs'
    "

    Change-Id: If1c06480c0adbd93ac8ed8b95c512b7355fd3098
    Signed-off-by: slin14 <email address hidden>

commit 2e2cdd7379687405d83e49e1fbef79e473068c24
Author: David Sullivan <email address hidden>
Date: Wed Sep 5 11:04:22 2018 -0400

    Add the starlingx-dashboard package to the iso

    This will need to be coordinated with the removal of the
    starlingx-dashboard content from horizon.

    Change-Id: I160e0edcf2074ca06c824739d329b76367de348d
    Signed-off-by: David Sullivan <email address hidden>
    Depends-On: https://review.openstack.org/599475
    Story: 2002891
    Task: 24821

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/603891

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/603891
Committed: https://git.openstack.org/cgit/openstack/stx-upstream/commit/?id=d981ba0b3a13f3e5d54fddf7ca5967726ee6a54d
Submitter: Zuul
Branch: master

commit 55eaec60182484b5c8f11cfef590a03290d487dc
Author: Eddie Ramirez <email address hidden>
Date: Mon Aug 27 11:31:21 2018 -0700

    Remove dc_admin from python-horizion

    dc_admin is now part of the starlingx-dashboard package. This change
    removes it from the python horizon package. Without this change the
    horizon package will fail to build.

    Change-Id: Ia1470ae0b110df3400bc69f15b6a8e6c52bf355e
    Signed-off-by: David Sullivan <email address hidden>
    Depends-On: https://review.openstack.org/599475
    Story: 2002891
    Task: 26220

commit 3fc7af31b8e5998741fad6b9b97905ce9f4eaedf
Author: Jack Ding <email address hidden>
Date: Thu Sep 6 10:22:55 2018 -0400

    Make dcorch-identity-api-proxy script executable

    The ocf script dcorch-identity-api-proxy was incorrectly set to 644, as
    a result distributed-cloud-services can not come up.
    Change permission of the script to 755 so that it is executable.

    Closes-Bug: 1790950
    Change-Id: I68e3067878fa6686dfd19db07018a42f05d92906
    Signed-off-by: Jack Ding <email address hidden>

commit a0be02a79a02838f793ebdf6a15dd8a23e5a7755
Author: slin14 <email address hidden>
Date: Wed Sep 5 06:49:39 2018 +0800

    remove non exist path to avoid error message when build srpm

    Story: 2003518
    Task: 26180

    It is to avoid error message like below:
    "
    ERROR: (1281): Bad path 'openstack/openstack-swift' in file '/localdisk/designer/slin14/starlingx/cgcs-root/stx/stx-upstream/centos_pkg_dirs'
    "

    Change-Id: If1c06480c0adbd93ac8ed8b95c512b7355fd3098
    Signed-off-by: slin14 <email address hidden>

commit 2e2cdd7379687405d83e49e1fbef79e473068c24
Author: David Sullivan <email address hidden>
Date: Wed Sep 5 11:04:22 2018 -0400

    Add the starlingx-dashboard package to the iso

    This will need to be coordinated with the removal of the
    starlingx-dashboard content from horizon.

    Change-Id: I160e0edcf2074ca06c824739d329b76367de348d
    Signed-off-by: David Sullivan <email address hidden>
    Depends-On: https://review.openstack.org/599475
    Story: 2002891
    Task: 24821