Trove-taskmanager fails to start if urls obtained from Keystone

[Mark Biciunas]

If urls for nova / cinder / swift are not specified in the config files, trove is supposed to resolve them against Keystone. However, trove-taskmanager fails start start with the following error:

Traceback (most recent call last):
  File "/usr/bin/trove-taskmanager", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/trove/cmd/common.py", line 65, in run
    return main_function(conf)
  File "/usr/lib/python2.7/site-packages/trove/cmd/taskmanager.py", line 34, in main
    startup(conf, None)
  File "/usr/lib/python2.7/site-packages/trove/cmd/taskmanager.py", line 27, in startup
    topic=topic)
  File "/usr/lib/python2.7/site-packages/trove/common/rpc/service.py", line 36, in __init__
    self.manager_impl = importutils.import_object(manager)
  File "/usr/lib/python2.7/site-packages/trove/openstack/common/importutils.py", line 38, in import_object
    return import_class(import_str)(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/trove/taskmanager/manager.py", line 44, in __init__
    context=self.admin_context)
  File "/usr/lib/python2.7/site-packages/trove/openstack/common/importutils.py", line 38, in import_object
    return import_class(import_str)(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/trove/extensions/mgmt/instances/models.py", line 234, in __init__
    self.nova_client = remote.create_admin_nova_client(self.context)
  File "/usr/lib/python2.7/site-packages/trove/common/remote.py", line 117, in create_admin_nova_client
    client = create_nova_client(context)
  File "/usr/lib/python2.7/site-packages/trove/common/remote.py", line 100, in nova_client
    endpoint_region=CONF.os_region_name)
  File "/usr/lib/python2.7/site-packages/trove/common/remote.py", line 56, in get_endpoint
    raise exception.EmptyCatalog()
EmptyCatalog: Empty catalog.

Supplying url information in the trove-taskmanager.conf file allows the service to start normaly.

[Denis M. (dmakogon)]

This doesn't seem to be a bug, since you have to supply correct OpenStack region.

[Sushil Kumar (sushil-kumar2)]

Agreed to Denis, there is the need of providing correct OpenStack region using os_region_name in the config files.

[Sushil Kumar (sushil-kumar2)]

Also for a devstack setup with more than one region, this is gonna be fixed using https://review.openstack.org/#/c/140121/

whereas if you have any other region applicable you can specify the same in configs.

[Mark Biciunas (mbiciunas)]

Thank you for responding.

However, os_region_name was specified in trove-taskmanager.conf and is a valid region. So this can't be the issue.

Also, there is only one region defined = RegionOne.

The following is the relevant section from trove-taskmanager.conf:

    # Configuration options for talking to nova via the novaclient.
    trove_auth_url = http://10.40.10.230:5000/v2.0
    #nova_compute_url = http://localhost:8774/v2
    #nova_compute_url = http://10.40.10.230:8774/v2
    #cinder_url = http://localhost:8776/v1
    #cinder_url = http://10.40.10.230:8776/v1
    #swift_url = http://localhost:8080/v1/AUTH_
    #swift_url = http://10.40.10.230:8080/v1/AUTH_
    #neutron_url = http://localhost:9696/

    # nova_compute_url, cinder_url, swift_url, and heat_url can all be fetched
    # from Keystone. To fetch from Keystone, comment out nova_compute_url,
    # cinder_url, swift_url, and heat_url and optionally uncomment the lines below.

    # Region name of this node. Used when searching catalog. Default value is None.
    os_region_name = RegionOne
    # Service type to use when searching catalog.
    nova_compute_service_type = compute
    # Service type to use when searching catalog.
    cinder_service_type = volumev2
    # Service type to use when searching catalog.
    swift_service_type = object-store
    # Service type to use when searching catalog.
    #heat_service_type = orchestration
    # Service type to use when searching catalog.
    #neutron_service_type = network

Some notes about our configuration:
    - Network is nova-network, no neutron.
    - Distribution is RedHat Openstack 5 running on Red Hat Linux 7
    - Installed using packstack
    - All services are on one machine, save for Trove which has been installed to a separate machine.

Finally, as per ther orginal bug filing, Trove works if the urls are specified. So the issue is strictly with url lookup.

I think the status on this bug should be changed back to New and properly investigated. If there is further information I can give you to help reproduce, please let me know.

[Javier Peña (jpena-c)]

I was hit by the same issue. As a workaround, setting "exists_notification_transformer = " (empty value) in trove-taskmanager.conf worked for me.

I'm quite sure I got the solution somewhere online, but I cannot find it now, so leaving it here in case it helps someone else.

[Doug Shelley (0-doug)]

The issue here is that the Task Manager sets up an exists notification transformer during it's startup sequence:

(from taskmanager/manager.py)
     self.admin_context = TroveContext(
            user=CONF.nova_proxy_admin_user,
            auth_token=CONF.nova_proxy_admin_pass,
            tenant=CONF.nova_proxy_admin_tenant_id)
        if CONF.exists_notification_transformer:
            self.exists_transformer = importutils.import_object(
                CONF.exists_notification_transformer,
                context=self.admin_context)

The admin_context hasn't gone through the process of authenticating so the service_catalog property isn't set. During the initialization of the NovaNotificationTransformer:
(from extensions/mgmt/instances/models.py)
class NovaNotificationTransformer(NotificationTransformer):
    def __init__(self, **kwargs):
        super(NovaNotificationTransformer, self).__init__(**kwargs)
        self.context = kwargs['context']
        self.nova_client = remote.create_admin_nova_client(self.context)
        self._flavor_cache = {}

If you don't set CONF.nova_compute_url, the call to create_admin_nova_client will attempt to use the context.service_catalog which in this case is never set.

In order for this code to work, I believe the Task Manager is going to have to authenticate the admin_context with Keystone to get the service_catalog.

(BTW, the workaround mentioned above works because this code path is not executed if exists_notification_transformer is None.)

[Amrith Kumar (amrith)]

Nikhil, any word on this?

[Shaik Apsar (sa709c)]

I'm done with the changes for the review and please review the changes.

Now trove-taskmanager service is able to startup without nova_compute_url and we don't need to be disable exists_notification_transformer as workaround.

Since we are using nova admin credentials with valid trove_auth_url, 'trove.extensions.mgmt.instances.models.NovaNotificationTransformer' can make use of trove.common.single_tenant_remote module to get the nova admin client.

[Chhavi Agarwal (chhagarw)]

Can you share the review for this bug, so I can have a look, I am also facing the similar issue

[Tomasz Nowak (tnowak)]

It's here: https://review.openstack.org/#/c/385584/

[Chhavi Agarwal (chhagarw)]

Checked the proposed fix, but this is not fixing the actual issue for EmptyCatalogue, rather its skipping the service catalogue, and using the single_remote_tenant.
To actually fix this issue, we need to use the Keystone Session while creating the TroveContext, and pass the actual auth_token, this will initialize the service_catalogue for Trove.
Similar approach is been used in openstack-heat project
https://github.com/openstack/heat/blob/master/heat/engine/clients/os/nova.py#L81

[OpenStack Infra (hudson-openstack)]

Change abandoned by amrith (<email address hidden>) on branch: master
Review: https://review.openstack.org/385584
Reason: abandoning for inactivity

[Manoj Kumar (manojnkumar)]

Chhavi, do you plan to drop the change you proposed?

[Takashi Kajinami (kajinamit)]

This looks like a bug in trove itself. So we have nothing we can do from deployment tool's perspective.