trove_auth_url is not populated properly everywhere

[Erik Olof Gunnar Andersson]

Starting with Trove 7.0.0rc2 there is an issue with certain configuration options not getting populated properly on startup.

This is caused by the following patch.
https://github.com/openstack/trove/commit/18f23a71777cdab1a397cafe0a677d9692eccacd

The reason for this is because task_api is imported before the configuration has been loaded, causing some variables to only use the default configuration value, e.g. trove/common/remotes.py

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/444040

[Erik Olof Gunnar Andersson (eandersson)]

This bug basically means that none of these values can be set to anything but the default value, e.g. trove_auth_url will never be respected and always fallback to localhost.

[Erik Olof Gunnar Andersson (eandersson)]

This depends on the order of imports, so trove-api may not be impacted by this, but trove-taskmanager would.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/455463

[Javier Castillo (javcasalc)]

Any workaround for this bug? It's not possible to deploy DB instances in ocata right now....

[Erik Olof Gunnar Andersson (eandersson)]

The only way we found around this was to manually patch the code with my suggested patch.

[Javier Castillo (javcasalc)]

I've applied your patch, but it still getting these logs:

2017-05-08 16:50:52.120 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-05-08 16:50:52.120 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming
2017-05-08 16:50:52.120 7 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2017-05-08 16:50:52.120 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 222, in dispatch
2017-05-08 16:50:52.120 7 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2017-05-08 16:50:52.120 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 192, in _do_dispatch
2017-05-08 16:50:52.120 7 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2017-05-08 16:50:52.120 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/osprofiler/profiler.py", line 153, in wrapper
2017-05-08 16:50:52.120 7 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2017-05-08 16:50:52.120 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/trove/taskmanager/manager.py", line 384, in create_instance
2017-05-08 16:50:52.120 7 ERROR oslo_messaging.rpc.server locality)
2017-05-08 16:50:52.120 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/trove/taskmanager/manager.py", line 364, in _create_instance
2017-05-08 16:50:52.120 7 ERROR oslo_messaging.rpc.server scheduler_hints)
2017-05-08 16:50:52.120 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/trove/taskmanager/models.py", line 483, in create_instance
2017-05-08 16:50:52.120 7 ERROR oslo_messaging.rpc.server self._log_and_raise(e, msg, err)
2017-05-08 16:50:52.120 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/trove/taskmanager/models.py", line 958, in _log_and_raise
2017-05-08 16:50:52.120 7 ERROR oslo_messaging.rpc.server raise TroveError(message=full_message)
2017-05-08 16:50:52.120 7 ERROR oslo_messaging.rpc.server TroveError: Error creating security group for instance: 2eeb6edc-fd7f-4805-b079-b17c0299b7a2
2017-05-08 16:50:52.120 7 ERROR oslo_messaging.rpc.server Unable to establish connection to http://0.0.0.0:5000/v2.0/tokens: HTTPConnectionPool(host='0.0.0.0', port=5000): Max retries exceeded with url: /v2.0/tokens (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f21339f3dd0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
2017-05-08 16:50:52.120 7 ERROR oslo_messaging.rpc.server

[Erik Olof Gunnar Andersson (eandersson)]

@javcasalc Did you patch the Trove Guest Agent as well?

[Javier Castillo (javcasalc)]

Patched API, TaskManager and Conductor. Guest Agente isn't running inside trove imagen (https://tarballs.openstack.org/trove/images/ubuntu/)??

Besides this, i ccanot even delete the failed instances.....

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/444040
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=db320b80a568f25f59da233853c530e647a0c496
Submitter: Jenkins
Branch: master

commit db320b80a568f25f59da233853c530e647a0c496
Author: Erik Olof Gunnar Andersson <email address hidden>
Date: Thu Mar 9 15:49:49 2017 -0800

    Fixing PROXY_AUTH_URL not being populated properly

    In a recent patch some imports were added to the top of
    cmd files, before the call to load the configuration
    was issued. This caused anything initilized before the
    configuration to fallback on default values.

    Because of this, for example PROXY_AUTH_URL found in
    trove/common/remote.py would always be set to the
    default value: "http://0.0.0.0:5000/v2.0".
    > PROXY_AUTH_URL = CONF.trove_auth_url

    This patch changes the config values in the
    trove/common/*remote.py files to be lazy loaded.

    Co-Authored-By: Riccardo Pittau <email address hidden>
    Change-Id: Ie3893680c406d92fe2b20e892c29fce675d7f224
    Closes-Bug: #1671655

[Javier Castillo (javcasalc)]

I only have patches for:

 trove/common/remote.py
 trove/common/single_tenant_remote.py
 trove/common/trove_remote.py

is there any other patch for Trove Guest Agent??

[Riccardo Pittau (rpittau)]

@javcasalc you need to apply the patch to the code inside the image you're using to spin up the trove instances

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/455463
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=2da1a2485b336ac83331b9dd465dde744665ecba
Submitter: Jenkins
Branch: stable/ocata

commit 2da1a2485b336ac83331b9dd465dde744665ecba
Author: Erik Olof Gunnar Andersson <email address hidden>
Date: Thu Mar 9 15:49:49 2017 -0800

    Fixing PROXY_AUTH_URL not being populated properly

    In a recent patch some imports were added to the top of
    cmd files, before the call to load the configuration
    was issued. This caused anything initilized before the
    configuration to fallback on default values.

    Because of this, for example PROXY_AUTH_URL found in
    trove/common/remote.py would always be set to the
    default value: "http://0.0.0.0:5000/v2.0".
    > PROXY_AUTH_URL = CONF.trove_auth_url

    This patch changes the config values in the
    trove/common/*remote.py files to be lazy loaded.

    Co-Authored-By: Riccardo Pittau <email address hidden>
    Change-Id: Ie3893680c406d92fe2b20e892c29fce675d7f224
    Closes-Bug: #1671655
    (cherry picked from commit db320b80a568f25f59da233853c530e647a0c496)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/trove 8.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/trove ocata-eol release.