Trove-taskmanager fails to start if urls obtained from Keystone

Bug #1402055 reported by Mark Biciunas on 2014-12-12
This bug affects 15 people
Affects Status Importance Assigned to Milestone
OpenStack DBaaS (Trove)
Chhavi Agarwal

Bug Description

If urls for nova / cinder / swift are not specified in the config files, trove is supposed to resolve them against Keystone. However, trove-taskmanager fails start start with the following error:

Traceback (most recent call last):
  File "/usr/bin/trove-taskmanager", line 10, in <module>
  File "/usr/lib/python2.7/site-packages/trove/cmd/", line 65, in run
    return main_function(conf)
  File "/usr/lib/python2.7/site-packages/trove/cmd/", line 34, in main
    startup(conf, None)
  File "/usr/lib/python2.7/site-packages/trove/cmd/", line 27, in startup
  File "/usr/lib/python2.7/site-packages/trove/common/rpc/", line 36, in __init__
    self.manager_impl = importutils.import_object(manager)
  File "/usr/lib/python2.7/site-packages/trove/openstack/common/", line 38, in import_object
    return import_class(import_str)(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/trove/taskmanager/", line 44, in __init__
  File "/usr/lib/python2.7/site-packages/trove/openstack/common/", line 38, in import_object
    return import_class(import_str)(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/trove/extensions/mgmt/instances/", line 234, in __init__
    self.nova_client = remote.create_admin_nova_client(self.context)
  File "/usr/lib/python2.7/site-packages/trove/common/", line 117, in create_admin_nova_client
    client = create_nova_client(context)
  File "/usr/lib/python2.7/site-packages/trove/common/", line 100, in nova_client
  File "/usr/lib/python2.7/site-packages/trove/common/", line 56, in get_endpoint
    raise exception.EmptyCatalog()
EmptyCatalog: Empty catalog.

Supplying url information in the trove-taskmanager.conf file allows the service to start normaly.

Denis M. (dmakogon) wrote :

This doesn't seem to be a bug, since you have to supply correct OpenStack region.

Sushil Kumar (sushil-kumar2) wrote :

Agreed to Denis, there is the need of providing correct OpenStack region using os_region_name in the config files.

Changed in trove:
status: New → Invalid
Sushil Kumar (sushil-kumar2) wrote :

Also for a devstack setup with more than one region, this is gonna be fixed using

whereas if you have any other region applicable you can specify the same in configs.

Mark Biciunas (mbiciunas) wrote :

Thank you for responding.

However, os_region_name was specified in trove-taskmanager.conf and is a valid region. So this can't be the issue.

Also, there is only one region defined = RegionOne.

The following is the relevant section from trove-taskmanager.conf:

    # Configuration options for talking to nova via the novaclient.
    trove_auth_url =
    #nova_compute_url = http://localhost:8774/v2
    #nova_compute_url =
    #cinder_url = http://localhost:8776/v1
    #cinder_url =
    #swift_url = http://localhost:8080/v1/AUTH_
    #swift_url =
    #neutron_url = http://localhost:9696/

    # nova_compute_url, cinder_url, swift_url, and heat_url can all be fetched
    # from Keystone. To fetch from Keystone, comment out nova_compute_url,
    # cinder_url, swift_url, and heat_url and optionally uncomment the lines below.

    # Region name of this node. Used when searching catalog. Default value is None.
    os_region_name = RegionOne
    # Service type to use when searching catalog.
    nova_compute_service_type = compute
    # Service type to use when searching catalog.
    cinder_service_type = volumev2
    # Service type to use when searching catalog.
    swift_service_type = object-store
    # Service type to use when searching catalog.
    #heat_service_type = orchestration
    # Service type to use when searching catalog.
    #neutron_service_type = network

Some notes about our configuration:
    - Network is nova-network, no neutron.
    - Distribution is RedHat Openstack 5 running on Red Hat Linux 7
    - Installed using packstack
    - All services are on one machine, save for Trove which has been installed to a separate machine.

Finally, as per ther orginal bug filing, Trove works if the urls are specified. So the issue is strictly with url lookup.

I think the status on this bug should be changed back to New and properly investigated. If there is further information I can give you to help reproduce, please let me know.

Changed in trove:
status: Invalid → New
Changed in trove:
importance: Undecided → Medium
assignee: nobody → Nikhil Manchanda (slicknik)
Javier Peña (jpena-c) wrote :

I was hit by the same issue. As a workaround, setting "exists_notification_transformer = " (empty value) in trove-taskmanager.conf worked for me.

I'm quite sure I got the solution somewhere online, but I cannot find it now, so leaving it here in case it helps someone else.

Changed in puppet-trove:
status: New → Triaged
Doug Shelley (0-doug) wrote :

The issue here is that the Task Manager sets up an exists notification transformer during it's startup sequence:

(from taskmanager/
     self.admin_context = TroveContext(
        if CONF.exists_notification_transformer:
            self.exists_transformer = importutils.import_object(

The admin_context hasn't gone through the process of authenticating so the service_catalog property isn't set. During the initialization of the NovaNotificationTransformer:
(from extensions/mgmt/instances/
class NovaNotificationTransformer(NotificationTransformer):
    def __init__(self, **kwargs):
        super(NovaNotificationTransformer, self).__init__(**kwargs)
        self.context = kwargs['context']
        self.nova_client = remote.create_admin_nova_client(self.context)
        self._flavor_cache = {}

If you don't set CONF.nova_compute_url, the call to create_admin_nova_client will attempt to use the context.service_catalog which in this case is never set.

In order for this code to work, I believe the Task Manager is going to have to authenticate the admin_context with Keystone to get the service_catalog.

(BTW, the workaround mentioned above works because this code path is not executed if exists_notification_transformer is None.)

Amrith Kumar (amrith) on 2015-10-21
Changed in trove:
status: New → Confirmed
Amrith Kumar (amrith) wrote :

Nikhil, any word on this?

Changed in trove:
milestone: none → newton-1
Amrith Kumar (amrith) on 2016-09-18
Changed in trove:
milestone: newton-1 → ongoing
Changed in trove:
assignee: Nikhil Manchanda (slicknik) → Shaik Apsar (sa709c)
status: Confirmed → In Progress
Shaik Apsar (sa709c) wrote :

I'm done with the changes for the review and please review the changes.

Now trove-taskmanager service is able to startup without nova_compute_url and we don't need to be disable exists_notification_transformer as workaround.

Since we are using nova admin credentials with valid trove_auth_url, 'trove.extensions.mgmt.instances.models.NovaNotificationTransformer' can make use of trove.common.single_tenant_remote module to get the nova admin client.

Chhavi Agarwal (chhagarw) wrote :

Can you share the review for this bug, so I can have a look, I am also facing the similar issue

Chhavi Agarwal (chhagarw) wrote :

Checked the proposed fix, but this is not fixing the actual issue for EmptyCatalogue, rather its skipping the service catalogue, and using the single_remote_tenant.
To actually fix this issue, we need to use the Keystone Session while creating the TroveContext, and pass the actual auth_token, this will initialize the service_catalogue for Trove.
Similar approach is been used in openstack-heat project

Change abandoned by amrith (<email address hidden>) on branch: master
Reason: abandoning for inactivity

Changed in trove:
assignee: Shaik Apsar (sa709c) → Chhavi Agarwal (chhagarw)
Manoj Kumar (manojnkumar) wrote :

Chhavi, do you plan to drop the change you proposed?

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers