This is hit with both Queens & Rocky branches
Request sent to create keystone endpoints, failed with error "An unexpected error prevented the server from fulfilling your request. (HTTP 500)"
TASK [create identity endpoints] ***********************************************
failed: [osctrl01] (item={u'url': u'http://60.0.25.189:5000/v3', u'type': u'public'}) => {"changed": true, "cmd": ["/opt/mhos/python/bin/openstack", "endpoint", "create", "--region", "SC2", "identity", "public", "http://60.0.25.189:5000/v3"], "delta": "0:00:00.610018", "end": "2018-12-01 07:15:56.894998", "failed": true, "item": {"type": "public", "url": "http://60.0.25.189:5000/v3"}, "rc": 1, "start": "2018-12-01 07:15:56.284980", "stderr": "An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-125826e3-7c54-4ee4-bb47-69c2a59e9d06)", "stdout": "", "stdout_lines": [], "warnings": []}
failed: [osctrl01] (item={u'url': u'http://60.0.25.188:5000/v3', u'type': u'internal'}) => {"changed": true, "cmd": ["/opt/mhos/python/bin/openstack", "endpoint", "create", "--region", "SC2", "identity", "internal", "http://60.0.25.188:5000/v3"], "delta": "0:00:00.741814", "end": "2018-12-01 07:15:57.787091", "failed": true, "item": {"type": "internal", "url": "http://60.0.25.188:5000/v3"}, "rc": 1, "start": "2018-12-01 07:15:57.045277", "stderr": "An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-ac52526c-3450-44d5-b13c-42d9040c2fc5)", "stdout": "", "stdout_lines": [], "warnings": []}
failed: [osctrl01] (item={u'url': u'http://60.0.25.188:35357/v3', u'type': u'admin'}) => {"changed": true, "cmd": ["/opt/mhos/python/bin/openstack", "endpoint", "create", "--region", "SC2", "identity", "admin", "http://60.0.25.188:35357/v3"], "delta": "0:00:00.599662", "end": "2018-12-01 07:15:58.543126", "failed": true, "item": {"type": "admin", "url": "http://60.0.25.188:35357/v3"}, "rc": 1, "start": "2018-12-01 07:15:57.943464", "stderr": "An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-629c6181-433b-4e98-9220-a8028f1dc17a)", "stdout": "", "stdout_lines": [], "warnings": []}
to retry, use: --limit @/tmp/ansible_os_run/3825619e-6adf-4a9e-a681-d03a0b7e8396/nsbu_cqe_openstack/initialize/site.retry
PLAY RECAP *********************************************************************
osctrl01 : ok=8 changed=5 unreachable=0 failed=1
osctrl02 : ok=3 changed=3 unreachable=0 failed=0
Keystone service unavailable server error:
TASK [create openstack id and add plugin] **************************************
changed: [osctrl01]
TASK [debug] *******************************************************************
ok: [osctrl01] => {
"openstack_token_id.stdout.split()[-2]": "gAAAAABcAi1-eWXyCq7q47S4CpE1PQQAD6xo7l-ngoI6D2Re58lymp0goziGxi2bOm3G5j5_dnBpWHE6pd-0okoETLVtc9w1bDo5LGD69BL7wIW69hbzsmJPnrKKTzdnV2MIEeNU49PfMILD6t_6SNpVMQ_BN5RULVK0tq20mJ09qiQUAy94hQg"
}
TASK [uri] *********************************************************************
fatal: [osctrl01]: FAILED! => {"cache_control": "no-cache", "changed": false, "connection": "close", "content": "<html><body><h1>503 Service Unavailable</h1>\nNo server is available to handle this request.\n</body></html>\n", "content_type": "text/html", "failed": true, "msg": "Status code was not [201]: HTTP Error 503: Service Unavailable", "redirected": false, "status": 503, "url": "http://60.0.25.188:9696/v2.0/project-plugin-maps"}
to retry, use: --limit @/tmp/ansible_os_run/3825619e-6adf-4a9e-a681-d03a0b7e8396/nsbu_cqe_openstack/initialize/site.retry
PLAY RECAP *********************************************************************
oscompute01 : ok=3 changed=2 unreachable=0 failed=0
oscompute02 : ok=2 changed=2 unreachable=0 failed=0
oscompute03 : ok=2 changed=2 unreachable=0 failed=0
osctrl01 : ok=33 changed=28 unreachable=0 failed=1
osctrl02 : ok=3 changed=3 unreachable=0 failed=0
oskvm01 : ok=3 changed=3 unreachable=0 failed=0
oskvm02 : ok=3 changed=3 unreachable=0 failed=0
2018-12-01 12:12:04,124 INFO Exit code:512
2018-12-01 12:12:04,125 WARNING Either install or initialize failed
2018-12-01 12:12:04,125 INFO Executing cmd:sudo ansible-playbook -i etc/tb201_nsxtvd_ansible_invt_hosts etc/uninstall_site.yml -e "host_key_checking=False ansible_user=root ansible_ssh_pass=stack building=True ansible_ssh_common_args='-o StrictHostKeyChecking=no'"
keystone log trace,
admin.log:2018-12-01 07:15:57.735 4580 DEBUG keystone.middleware.auth [req-125826e3-7c54-4ee4-bb47-69c2a59e9d06 - - - - -] RBAC: auth_context: {} fill_context /opt/mhos/openstack/keystone/lib/python2.7/site-packages/keystone/middleware/auth.py:247
admin.log:2018-12-01 07:15:57.736 4580 INFO keystone.common.wsgi [req-125826e3-7c54-4ee4-bb47-69c2a59e9d06 - - - - -] POST http://60.0.25.188:35357/v3/endpoints
admin.log:2018-12-01 07:15:57.737 4580 WARNING keystone.common.controller [req-125826e3-7c54-4ee4-bb47-69c2a59e9d06 - - - - -] RBAC: Bypassing authorization
admin.log:2018-12-01 07:15:57.749 4580 ERROR keystone.common.wsgi [req-125826e3-7c54-4ee4-bb47-69c2a59e9d06 - - - - -] (pymysql.err.IntegrityError) (1452, u'Cannot add or update a child row: a foreign key constraint fails (`keystone`.`endpoint`, CONSTRAINT `fk_endpoint_region_id` FOREIGN KEY (`region_id`) REFERENCES `region` (`id`))') [SQL: u'INSERT INTO endpoint (id, legacy_endpoint_id, interface, region_id, service_id, url, enabled, extra) VALUES (%(id)s, %(legacy_endpoint_id)s, %(interface)s, %(region_id)s, %(service_id)s, %(url)s, %(enabled)s, %(extra)s)'] [parameters: {'region_id': u'SC2', 'extra': '{}', 'url': u'http://60.0.25.189:5000/v3', 'enabled': 1, 'legacy_endpoint_id': None, 'interface': u'public', 'service_id': u'bc2f72f76d3a44cf81cd05bc581c1cda', 'id': 'dea654331d4644b89689ec452edb9fed'}] (Background on this error at: http://sqlalche.me/e/gkpj): DBReferenceError: (pymysql.err.IntegrityError) (1452, u'Cannot add or update a child row: a foreign key constraint fails (`keystone`.`endpoint`, CONSTRAINT `fk_endpoint_region_id` FOREIGN KEY (`region_id`) REFERENCES `region` (`id`))') [SQL: u'INSERT INTO endpoint (id, legacy_endpoint_id, interface, region_id, service_id, url, enabled, extra) VALUES (%(id)s, %(legacy_endpoint_id)s, %(interface)s, %(region_id)s, %(service_id)s, %(url)s, %(enabled)s, %(extra)s)'] [parameters: {'region_id': u'SC2', 'extra': '{}', 'url': u'http://60.0.25.189:5000/v3', 'enabled': 1, 'legacy_endpoint_id': None, 'interface': u'public', 'service_id': u'bc2f72f76d3a44cf81cd05bc581c1cda', 'id': 'dea654331d4644b89689ec452edb9fed'}] (Background on this error at: http://sqlalche.me/e/gkpj)
Does the region "SC2" exist already?