Original issue reported in https://bugzilla.redhat.com/show_bug.cgi?id=1703045
When deploying Octavia services on networker nodes that don't have access to public endpoints (asymmetric routing, spine-leaf network topology), overcloud deployment fails because octavia deployment scripts use public endpoints to create neutron networks, to set quotas, etc...
Issue was reported in queens and rocky releases and reproduced in stein release.
Steps to reproduce:
1. Spine-Leaf network topology + OVN + Octavia + 2 Networker nodes
2. Update roles configuration in roles_data.yaml to run octavia services on Networker
- name: Controller0
...
ServicesDefault:
...
- OS::TripleO::Services::OctaviaApi
...
- name: Networker0
...
ServicesDefault:
...
- OS::TripleO::Services::OctaviaDeploymentConfig
- OS::TripleO::Services::OctaviaHealthManager
- OS::TripleO::Services::OctaviaHousekeeping
- OS::TripleO::Services::OctaviaWorker
Actual results:
Deployment failed with error in mistral:
overcloud.AllNodesDeploySteps.WorkflowTasks_Step5_Execution:
resource_type: OS::TripleO::WorkflowSteps
physical_resource_id: c1120606-31b9-47bc-8cbf-d543beb08e1e
status: CREATE_FAILED
status_reason: |
resources.WorkflowTasks_Step5_Execution: ERROR
More detailes about Octavia failure
2019-06-06 15:56:02.263 31313 INFO workflow_trace [req-fd58419e-01c7-49bc-93ba-af2f1fd87dac 78f7841a1c30427dbee402e4fd056f55 d70b2b1ee37b4acf8b250
e047e8db948 - default default] Workflow 'tripleo.overcloud.workflow_tasks.step5' [RUNNING -> ERROR, msg=Failure caused by error in tasks: octavia_
post_workflow
octavia_post_workflow [task_ex_id=9b26f380-741d-4995-9d8e-1f2cc6e24133] -> Failure caused by error in tasks: config_octavia
config_octavia [task_ex_id=29b1a410-2566-4a53-a64b-c0c5b0f5be81] -> Failed to run action [action_ex_id=6710acf0-b9fe-435d-beb4-9ba4e132a961, act
ion_cls='<class 'mistral.actions.action_factory.AnsiblePlaybookAction'>', attributes='{}', params='{u'remote_user': u'tripleo-admin', u'ssh_extra_
args': u'-o UserKnownHostsFile=/dev/null', u'inventory': {u'octavia_nodes': {u'hosts': {u'192.168.24.20': {}, u'192.168.24.36': {}}}}, u'become_us
er': u'root', u'extra_vars': {u'client_cert_path': u'/etc/octavia/certs/client.pem', u'mgmt_port_dev': u'o-hm0', u'lb_mgmt_subnet_name': u'lb-mgmt
-subnet', u'lb_mgmt_net_name': u'lb-mgmt-net', u'os_project_name': u'admin', u'os_username': u'admin', u'os_identity_api_version': u'3', u'amp_ima
ge_tag': u'amphora-image', u'ca_passphrase': u'mpTCUTuw7Kr3WTzGJX8grWsMr', u'auth_project_name': u'service', u'lb_mgmt_subnet_gateway': u'172.24.0
.1', u'lb_mgmt_subnet_pool_start': u'172.24.0.2', u'lb_mgmt_subnet_pool_end': u'172.24.255.254', u'os_auth_url': u'https://10.0.10.100:13000/v3',
u'ca_private_key_path': u'/etc/octavia/certs/private/cakey.pem', u'lb_sec_group_name': u'lb-mgmt-subnet', u'os_password': u'xUKFxkHwwqUkRgxAecCg4T
Qvw', u'os_auth_type': u'password', u'lb_mgmt_subnet_cidr': u'172.24.0.0/16', u'ca_cert_path': u'/etc/octavia/certs/ca_01.pem', u'generate_certs':
True}, u'verbosity': 0, u'extra_env_variables': {u'ANSIBLE_SSH_RETRIES': u'3', u'ANSIBLE_HOST_KEY_CHECKING': u'False'}, u'ssh_private_key': u'---
--BEGIN RSA PRIVATE KEY-----\nMIIEowIBAAKCAQEA3ZA5ubbSQhIybnX1GaDWeqa4XEeo8rYgCNseSdB/rfw19ZJ7\nLqjopsadqcH8IrIdFN0MBWRlcYe0BAxP3O7C1CU5+Yl/RTa9cV
XP9S8m2WEhhwYY\np/wxLnxzFRaCjMWolRwuQZb+j/JSd6c5Z/+en502lVmybFv/QYXredT0ZtwG5GJb\nuLZ9LqVLYaGDZ/y9BWGZDztwVjWBmWlQG5PSU+MAgpjxNtj4AlGCVqHNRdnvLS4e
\nXw+XJXDs5IQMU+yFrqrNP8rVvW2CJwjJs42dN70QyFDUIMvRkYS+vDCOWFp9L5CI\nyjfEFZOir8O93hR4Yp3kY2kzr+6XZt5+yfl/qwIDAQABAoIBADTolgBVOgxxD/30\nyRzfnZgYa/oN
Wrjq6Od0e90gnvzLN4929VeFGlmJIGlDW5RleDBdQNugx+C+iSxW\nTFPz6C6E3T1Lqkq68a440bo5EkviuADoYcbroEr7iPfGMlKveLxyyFD9X7i8IxlT\no4/EKPrwrfIoQ0VBCsl402x6gA
p3mpyi5sUa2d2xDkIs4xmJ+qvHLKzRqg6YrNTu\n5FCRRhsARw/IcJXxLmMu0g67i9O1kMUGbs6XX8x5JMBigNY+i6RS97JTIu+ddXpv\njT7pOOVwY0EymWyDIKyb9pPnsuJjYw2JFaDEANU/
zpv6Vd6ojjpx32RSuqPWNZzI\nYhrEEGECgYEA9JSZkHPk9EPVLdPhUP9BETKUgto2/m7jpP51QRywS1OXT4AUzPFS\nN+uKsufKZ0meK0/LB35HP1Jet7OJeJShpXPMbG2sCWR13fGsiR4NZZ
8iW6gy6kIJ\nN2BBAHzDLRzFFdLkzq30BwfNMWNoIMp5PQsxmAkLSABGBgkOVOI4+hkCgYEA5+iC\nTS6921oQVkuUusswhy+V0pV9guD2AWqzyNy1KIOLa6bShbmrtrGRPAYG5HGHbbb9\n73
BeXuTBP8yFb2NOYNqLNvTRZ7OucEpY0oVdR65v+DMHZR/jVM6NEn4jfrVIClYX\nvCyHwsqRFe1NreBNIAHvdT6OfNYV4T3qWrsgCGMCgYBKVELbKK2DIn5OAB9wqzJO\nFK4XmlOuPWsHgKGH
2T0ml0/bxFQN+KUBA59SQak8fJ4KEaTlMRZcAx9v+qsjrx/1\nFV0h8q6e6B3+Bm1l+nEd2h/p9RMMKGd+ocz/Zes28ZBf0ojg2vLXlCJjCQ/jL0Vr\nLNS0nMMF7bdaLDRjzaB9OQKBgA8ZYV
p7H5tnisbDlwRudFNo8r1KRGjAEuRWuSvr\nytO/dNVmgDB6vUZg207oKYy4I5QuJOxxCYPuKvLncwykj5bYw9WpLPUuir3+6TeT\nvVYMcnfbgrC/2cJMzHyWv+LhFLavkk4LLC+vlrCxyav3
fa4G0jt0/jv8iGIo8NhF\ndLl3AoGBANVn9ySPH9JnyCgUhaTIJtJbL3CDGwmH8VdDrx4hgmML5/7d53cQDEX3\n7ml3GpCsfv+vWB/QgC9q0iMOL8reuNaaT1Unj+52v+jbD6GVNsx+54SrlI
1zF11d\nYYKn3NBgJI9XkIr5NK1VE48xiopCdVM5DNClC8YjJYDPb7Xp5JKt\n-----END RSA PRIVATE KEY-----\n', u'become': True, u'ssh_common_args': u'-o StrictHo
stKeyChecking=no', u'playbook': u'/usr/share/tripleo-common/playbooks/octavia-files.yaml'}']
Unexpected error while running command.
TASK [octavia-overcloud-config : include_tasks] ********************************\nThursday 06 June 2019 15:55:57 -0400 (0:00:03.675) 0:00:03.754 ********* \nincluded: /usr/share/open
stack-tripleo-common/playbooks/roles/octavia-overcloud-config/tasks/network.yml for 192.168.24.20\n\nTASK [octavia-overcloud-config : create manag
ement network for load balancers] ***\nThursday 06 June 2019 15:55:57 -0400 (0:00:00.047) 0:00:03.802 ********* \nfatal: [192.168.24.20]: F
AILED! => {"changed": false, "cmd": "if [[ $(openstack network show lb-mgmt-net > /dev/null; echo $?) -eq 1 ]]; then\\n openstack network create -
f value -c id lb-mgmt-net\\n fi", "delta": "0:00:01.964978", "end": "2019-06-06 19:55:59.512877", "failed": true, "msg": "non-zero return code", "
rc": 1, "start": "2019-06-06 19:55:57.547899", "stderr": "Failed to discover available identity versions when contacting https://10.0.10.100:13000
/v3. Attempting to parse version from URL.\\nUnable to establish connection to https://10.0.10.100:13000/v3/auth/tokens: HTTPSConnectionPool(host=
\'10.0.10.100\', port=13000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError(\'<requests.packages.urllib3.connection
.VerifiedHTTPSConnection object at 0x7f6e2aff7910>: Failed to establish a new connection: [Errno 101] Network is unreachable\',))\\nFailed to disc
over available identity versions when contacting https://10.0.10.100:13000/v3. Attempting to parse version from URL.\\nUnable to establish connect
ion to https://10.0.10.100:13000/v3/auth/tokens: HTTPSConnectionPool(host=\'10.0.10.100\', port=13000): Max retries exceeded with url: /v3/auth/to
kens (Caused by NewConnectionError(\'<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f894e7f4910>: Failed to establish
a new connection: [Errno 101] Network is unreachable\',))", "stderr_lines": ["Failed to discover available identity versions when contacting https
://10.0.10.100:13000/v3. Attempting to parse version from URL.", "Unable to establish connection to https://10.0.10.100:13000/v3/auth/tokens: HTTP
SConnectionPool(host=\'10.0.10.100\', port=13000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError(\'<requests.packag
es.urllib3.connection.VerifiedHTTPSConnection object at 0x7f6e2aff7910>: Failed to establish a new connection: [Errno 101] Network is unreachable\
',))", "Failed to discover available identity versions when contacting https://10.0.10.100:13000/v3. Attempting to parse version from URL.", "Unab
le to establish connection to https://10.0.10.100:13000/v3/auth/tokens: HTTPSConnectionPool(host=\'10.0.10.100\', port=13000): Max retries exceede
d with url: /v3/auth/tokens (Caused by NewConnectionError(\'<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f894e7f4910
>: Failed to establish a new connection: [Errno 101] Network is unreachable\',))"], "stdout": "", "stdout_lines": []}\n\nPLAY RECAP **************
*******************************************************\n192.168.24.20 : ok=2 changed=0 unreachable=0 failed=1 \n\nThursda
y 06 June 2019 15:55:59 -0400 (0:00:02.385) 0:00:06.188 ********* \n=======================================================================
======== \n'
Stderr: u' [WARNING]: Could not match supplied host pattern, ignoring: undercloud\n'
2019-06-06 15:55:59.598 31393 ERROR mistral.executors.default_executor Traceback (most recent call last):
2019-06-06 15:55:59.598 31393 ERROR mistral.executors.default_executor File "/usr/lib/python2.7/site-packages/mistral/executors/default_executor
.py", line 114, in run_action
2019-06-06 15:55:59.598 31393 ERROR mistral.executors.default_executor result = action.run(action_ctx)
2019-06-06 15:55:59.598 31393 ERROR mistral.executors.default_executor File "/usr/lib/python2.7/site-packages/tripleo_common/actions/ansible.py"
, line 541, in run
2019-06-06 15:55:59.598 31393 ERROR mistral.executors.default_executor log_errors=processutils.LogErrors.ALL)
2019-06-06 15:55:59.598 31393 ERROR mistral.executors.default_executor File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py",
line 424, in execute
2019-06-06 15:55:59.598 31393 ERROR mistral.executors.default_executor cmd=sanitized_cmd)
2019-06-06 15:55:59.598 31393 ERROR mistral.executors.default_executor ProcessExecutionError: Unexpected error while running command.
2019-06-06 15:55:59.598 31393 ERROR mistral.executors.default_executor Command: ansible-playbook /usr/share/tripleo-common/playbooks/octavia-files:
Fix proposed to branch: master
Review: https:/