Hello,
I've just deployed trove using the ocata/stable openstack ansible playbooks (non AIO). Unfortunately, I'm running into the following error where I can't spawn database instances. The process gets to the trove task manager, creates a volume, but then fails when trying to talk to nova:
2017-03-13 12:23:02.169 18636 DEBUG trove.instance.models [-] Instance e3faa24a-50c4-42f8-bfc7-32c061b58718 service status is new. load_instance /openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/trove/instance/models.py:538
2017-03-13 12:23:02.179 18636 INFO trove.taskmanager.models [-] Creating instance e3faa24a-50c4-42f8-bfc7-32c061b58718.
2017-03-13 12:23:02.186 18636 DEBUG trove.taskmanager.models [-] Begin _create_server_volume_individually for id: e3faa24a-50c4-42f8-bfc7-32c061b58718 _create_server_volume_individually /openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/trove/taskmanager/models.py:897
2017-03-13 12:23:02.187 18636 DEBUG trove.taskmanager.models [-] trove volume support = True _build_volume_info /openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/trove/taskmanager/models.py:926
2017-03-13 12:23:02.187 18636 DEBUG trove.taskmanager.models [-] Begin _create_volume for id: e3faa24a-50c4-42f8-bfc7-32c061b58718 _create_volume /openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/trove/taskmanager/models.py:961
2017-03-13 12:23:04.072 18636 DEBUG trove.db.models [-] Saving DBInstance: {u'region_id': u'RegionOne', u'cluster_id': None, u'shard_id': None, u'deleted_at': None, u'id': u'e3faa24a-50c4-42f8-bfc7-32c061b58718', u'datastore_version_id': u'1234dec8-0aed-4952-a4e8-bedb03ed7dbc', 'errors': {}, u'hostname': None, u'server_status': None, u'task_description': u'The instance is building.', u'volume_size': 1, u'type': None, u'updated': datetime.datetime(2017, 3, 13, 16, 23, 4, 71339), '_sa_instance_state': <sqlalchemy.orm.state.InstanceState object at 0x7fb4e7af6910>, u'encrypted_key': '***', u'deleted': 0, u'configuration_id': None, u'volume_id': u'566b36f0-57ff-4957-8968-a8938eb28eb7', u'slave_of_id': None, u'task_start_time': None, u'name': u'testtrove', u'task_id': 5, u'created': datetime.datetime(2017, 3, 13, 16, 23, 2), u'tenant_id': u'5baf270fa98e43dca59e28e188afca9d', u'compute_instance_id': None, u'flavor_id': u'f39920d9-45ad-4377-bdaf-31ad42719290'} save /openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/trove/db/models.py:64
2017-03-13 12:23:04.714 18636 DEBUG trove.taskmanager.models [-] End _create_volume for id: e3faa24a-50c4-42f8-bfc7-32c061b58718 _create_volume /openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/trove/taskmanager/models.py:981
2017-03-13 12:23:04.714 18636 DEBUG trove.taskmanager.models [-] Created volume <Volume: 566b36f0-57ff-4957-8968-a8938eb28eb7> _build_volume /openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/trove/taskmanager/models.py:985
2017-03-13 12:23:04.715 18636 DEBUG trove.taskmanager.models [-] block_device = {'vdb': u'566b36f0-57ff-4957-8968-a8938eb28eb7::1:1'}
volume = [{'id': u'566b36f0-57ff-4957-8968-a8938eb28eb7', 'size': 1}]
device_path = /dev/vdb
mount_point = /var/lib/mysql _build_volume /openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/trove/taskmanager/models.py:1005
2017-03-13 12:23:04.717 18636 WARNING novaclient.v2.client [-] Property `management_url` is deprecated for SessionClient. It should be set via `endpoint_override` variable while class initialization.
2017-03-13 12:23:04.761 18636 WARNING keystoneauth.identity.generic.base [-] Discovering versions from the identity service failed when creating the password plugin. Attempting to determine version from URL.
2017-03-13 12:23:04.764 18636 ERROR trove.taskmanager.models [-] Failed to create server for instance e3faa24a-50c4-42f8-bfc7-32c061b58718
Unable to establish connection to http://0.0.0.0:5000/v2.0/tokens: HTTPConnectionPool(host='0.0.0.0', port=5000): Max retries exceeded with url: /v2.0/tokens (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fb4e7a7a0d0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
Traceback (most recent call last):
File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/trove/taskmanager/models.py", line 908, in _create_server_volume_individually
scheduler_hints)
File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/trove/taskmanager/models.py", line 1035, in _create_server
config_drive=config_drive, scheduler_hints=scheduler_hints)
File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/novaclient/v2/servers.py", line 1416, in create
**boot_kwargs)
File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/novaclient/v2/servers.py", line 779, in _boot
return_raw=return_raw, **kwargs)
File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/novaclient/base.py", line 361, in _create
resp, body = self.api.client.post(url, body=body)
File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 223, in post
return self.request(url, 'POST', **kwargs)
File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/novaclient/client.py", line 74, in request
**kwargs)
File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 374, in request
resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 142, in request
return self.session.request(url, method, **kwargs)
File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
return wrapped(*args, **kwargs)
File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/keystoneauth1/session.py", line 491, in request
auth_headers = self.get_auth_headers(auth)
File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/keystoneauth1/session.py", line 818, in get_auth_headers
return auth.get_headers(self, **kwargs)
File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/keystoneauth1/plugin.py", line 90, in get_headers
token = self.get_token(session)
File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 90, in get_token
return self.get_access(session).auth_token
File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 136, in get_access
self.auth_ref = self.get_auth_ref(session)
File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 198, in get_auth_ref
return self._plugin.get_auth_ref(session, **kwargs)
File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/keystoneauth1/identity/v2.py", line 65, in get_auth_ref
authenticated=False, log=False)
File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/keystoneauth1/session.py", line 766, in post
return self.request(url, 'POST', **kwargs)
File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
return wrapped(*args, **kwargs)
File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/keystoneauth1/session.py", line 616, in request
resp = send(**kwargs)
File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/keystoneauth1/session.py", line 690, in _send_request
raise exceptions.ConnectFailure(msg)
ConnectFailure: Unable to establish connection to http://0.0.0.0:5000/v2.0/tokens: HTTPConnectionPool(host='0.0.0.0', port=5000): Max retries exceeded with url: /v2.0/tokens (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fb4e7a7a0d0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2017-03-13 12:23:04.772 18636 DEBUG trove.db.models [-] Saving DBInstance: {u'region_id': u'RegionOne', u'cluster_id': None, u'shard_id': None, u'deleted_at': None, u'id': u'e3faa24a-50c4-42f8-bfc7-32c061b58718', u'datastore_version_id': u'1234dec8-0aed-4952-a4e8-bedb03ed7dbc', 'errors': {}, u'hostname': None, u'server_status': None, u'task_description': u'Build error: Server.', u'volume_size': 1, u'type': None, u'updated': datetime.datetime(2017, 3, 13, 16, 23, 4, 771405), '_sa_instance_state': <sqlalchemy.orm.state.InstanceState object at 0x7fb4e7a5e050>, u'encrypted_key': '***', u'deleted': 0, u'configuration_id': None, u'volume_id': u'566b36f0-57ff-4957-8968-a8938eb28eb7', u'slave_of_id': None, u'task_start_time': None, u'name': u'testtrove', u'task_id': 81, u'created': datetime.datetime(2017, 3, 13, 16, 23, 2), u'tenant_id': u'5baf270fa98e43dca59e28e188afca9d', u'compute_instance_id': None, u'flavor_id': u'f39920d9-45ad-4377-bdaf-31ad42719290'} save /openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/trove/db/models.py:64
2017-03-13 12:23:04.785 18636 DEBUG trove.common.notification [-] Sending event: dbaas.instance_create.error, {'client_ip': u'10.0.2.10', 'server_type': u'taskmanager', 'tenant_id': u'5baf270fa98e43dca59e28e188afca9d', 'exception': ['Traceback (most recent call last):\n', ' File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/trove/taskmanager/manager.py", line 384, in create_instance\n locality)\n', ' File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/trove/taskmanager/manager.py", line 364, in _create_instance\n scheduler_hints)\n', ' File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/trove/taskmanager/models.py", line 526, in create_instance\n scheduler_hints)\n', ' File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/trove/taskmanager/models.py", line 915, in _create_server_volume_individually\n self._log_and_raise(e, msg, err)\n', ' File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/trove/taskmanager/models.py", line 958, in _log_and_raise\n raise TroveError(message=full_message)\n', "TroveError: Failed to create server for instance e3faa24a-50c4-42f8-bfc7-32c061b58718\nUnable to establish connection to http://0.0.0.0:5000/v2.0/tokens: HTTPConnectionPool(host='0.0.0.0', port=5000): Max retries exceeded with url: /v2.0/tokens (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fb4e7a7a0d0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))\n"], 'instance_id': u'e3faa24a-50c4-42f8-bfc7-32c061b58718', 'server_ip': u'10.0.2.250:8779', 'request_id': u'req-f62c18f0-1adb-4a71-9087-d4ba2f0d3348', 'message': "Failed to create server for instance e3faa24a-50c4-42f8-bfc7-32c061b58718\nUnable to establish connection to http://0.0.0.0:5000/v2.0/tokens: HTTPConnectionPool(host='0.0.0.0', port=5000): Max retries exceeded with url: /v2.0/tokens (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fb4e7a7a0d0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))"} _notify /openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/trove/common/notification.py:405
2017-03-13 12:23:04.799 18636 DEBUG trove.db.models [-] Saving DBInstanceFault: {u'updated': datetime.datetime(2017, 3, 13, 16, 23, 4, 798510), '_sa_instance_state': <sqlalchemy.orm.state.InstanceState object at 0x7fb4e7a5ed50>, u'created': datetime.datetime(2017, 3, 13, 16, 23, 4, 798355), u'deleted': False, u'instance_id': u'e3faa24a-50c4-42f8-bfc7-32c061b58718', 'errors': {}, u'details': u'Server type: taskmanager\nTraceback (most recent call last):\n File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/trove/taskmanager/manager.py", line 384, in create_instance\n locality)\n File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/trove/taskmanager/manager.py", line 364, in _create_instance\n scheduler_hints)\n File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/trove/taskmanager/models.py", line 526, in create_instance\n scheduler_hints)\n File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/trove/taskmanager/models.py", line 915, in _create_server_volume_individually\n self._log_and_raise(e, msg, err)\n File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/trove/taskmanager/models.py", line 958, in _log_and_raise\n raise TroveError(message=full_message)\nTroveError:\n Failed to create server for instance e3faa24a-50c4-42f8-bfc7-32c061b58718\nUnable to establish connection to http://0.0.0.0:5000/v2.0/tokens:\n HTTPConnectionPool(host=\'0.0.0.0\', port=5000):\n Max retries exceeded with url: /v2.0/tokens\n(Caused by NewConnectionError(\'<requests.packages.urllib3.connection.HTTPConnection object at 0x7fb4e7a7a0d0>:\n Failed to establish a new connection: [Errno 111] ECONNREFUSED\',))\n', u'message': u"Failed to create server for instance e3faa24a-50c4-42f8-bfc7-32c061b58718\nUnable to establish connection to http://0.0.0.0:5000/v2.0/tokens:\n HTTPConnectionPool(host='0.0.0.0', port=5000):\n Max retries exceeded with url: /v2.0/tokens\n(Caused by N...", u'id': u'bcc57192-0502-4de7-ab49-09f50a2c4954'} save /openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/trove/db/models.py:64
2017-03-13 12:23:04.812 18636 ERROR oslo_messaging.rpc.server [-] Exception during message handling
2017-03-13 12:23:04.812 18636 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-03-13 12:23:04.812 18636 ERROR oslo_messaging.rpc.server File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming
2017-03-13 12:23:04.812 18636 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2017-03-13 12:23:04.812 18636 ERROR oslo_messaging.rpc.server File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 222, in dispatch
2017-03-13 12:23:04.812 18636 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2017-03-13 12:23:04.812 18636 ERROR oslo_messaging.rpc.server File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 192, in _do_dispatch
2017-03-13 12:23:04.812 18636 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2017-03-13 12:23:04.812 18636 ERROR oslo_messaging.rpc.server File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/osprofiler/profiler.py", line 153, in wrapper
2017-03-13 12:23:04.812 18636 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2017-03-13 12:23:04.812 18636 ERROR oslo_messaging.rpc.server File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/trove/taskmanager/manager.py", line 384, in create_instance
2017-03-13 12:23:04.812 18636 ERROR oslo_messaging.rpc.server locality)
2017-03-13 12:23:04.812 18636 ERROR oslo_messaging.rpc.server File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/trove/taskmanager/manager.py", line 364, in _create_instance
2017-03-13 12:23:04.812 18636 ERROR oslo_messaging.rpc.server scheduler_hints)
2017-03-13 12:23:04.812 18636 ERROR oslo_messaging.rpc.server File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/trove/taskmanager/models.py", line 526, in create_instance
2017-03-13 12:23:04.812 18636 ERROR oslo_messaging.rpc.server scheduler_hints)
2017-03-13 12:23:04.812 18636 ERROR oslo_messaging.rpc.server File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/trove/taskmanager/models.py", line 915, in _create_server_volume_individually
2017-03-13 12:23:04.812 18636 ERROR oslo_messaging.rpc.server self._log_and_raise(e, msg, err)
2017-03-13 12:23:04.812 18636 ERROR oslo_messaging.rpc.server File "/openstack/venvs/trove-15.0.0/lib/python2.7/site-packages/trove/taskmanager/models.py", line 958, in _log_and_raise
2017-03-13 12:23:04.812 18636 ERROR oslo_messaging.rpc.server raise TroveError(message=full_message)
2017-03-13 12:23:04.812 18636 ERROR oslo_messaging.rpc.server TroveError: Failed to create server for instance e3faa24a-50c4-42f8-bfc7-32c061b58718
2017-03-13 12:23:04.812 18636 ERROR oslo_messaging.rpc.server Unable to establish connection to http://0.0.0.0:5000/v2.0/tokens: HTTPConnectionPool(host='0.0.0.0', port=5000): Max retries exceeded with url: /v2.0/tokens (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fb4e7a7a0d0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2017-03-13 12:23:04.812 18636 ERROR oslo_messaging.rpc.server
My config for nova proxy from trove-taskmanager.conf is below:
# Nova
nova_compute_url = http://10.0.2.250:8774/v2.1
nova_proxy_admin_user = admin
nova_proxy_admin_pass = <redacted>
nova_proxy_admin_tenant_name = service
I'm note sure why it's trying to connect to 0.0.0.0 or trying to use v2 auth for keystone. However, this is preventing me from spawning database instances
This is blocking instance creation. Are you sure it's due to the same upstream bug?