Trove-taskmanager fails to start if urls obtained from Keystone

Bug #1402055 reported by Mark Biciunas on 2014-12-12
72
This bug affects 15 people
Affects Status Importance Assigned to Milestone
OpenStack DBaaS (Trove)
Medium
Chhavi Agarwal
puppet-trove
Undecided
Unassigned

Bug Description

If urls for nova / cinder / swift are not specified in the config files, trove is supposed to resolve them against Keystone. However, trove-taskmanager fails start start with the following error:

Traceback (most recent call last):
  File "/usr/bin/trove-taskmanager", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/trove/cmd/common.py", line 65, in run
    return main_function(conf)
  File "/usr/lib/python2.7/site-packages/trove/cmd/taskmanager.py", line 34, in main
    startup(conf, None)
  File "/usr/lib/python2.7/site-packages/trove/cmd/taskmanager.py", line 27, in startup
    topic=topic)
  File "/usr/lib/python2.7/site-packages/trove/common/rpc/service.py", line 36, in __init__
    self.manager_impl = importutils.import_object(manager)
  File "/usr/lib/python2.7/site-packages/trove/openstack/common/importutils.py", line 38, in import_object
    return import_class(import_str)(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/trove/taskmanager/manager.py", line 44, in __init__
    context=self.admin_context)
  File "/usr/lib/python2.7/site-packages/trove/openstack/common/importutils.py", line 38, in import_object
    return import_class(import_str)(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/trove/extensions/mgmt/instances/models.py", line 234, in __init__
    self.nova_client = remote.create_admin_nova_client(self.context)
  File "/usr/lib/python2.7/site-packages/trove/common/remote.py", line 117, in create_admin_nova_client
    client = create_nova_client(context)
  File "/usr/lib/python2.7/site-packages/trove/common/remote.py", line 100, in nova_client
    endpoint_region=CONF.os_region_name)
  File "/usr/lib/python2.7/site-packages/trove/common/remote.py", line 56, in get_endpoint
    raise exception.EmptyCatalog()
EmptyCatalog: Empty catalog.

Supplying url information in the trove-taskmanager.conf file allows the service to start normaly.

Denis M. (dmakogon) wrote :

This doesn't seem to be a bug, since you have to supply correct OpenStack region.

Sushil Kumar (sushil-kumar2) wrote :

Agreed to Denis, there is the need of providing correct OpenStack region using os_region_name in the config files.

Changed in trove:
status: New → Invalid
Sushil Kumar (sushil-kumar2) wrote :

Also for a devstack setup with more than one region, this is gonna be fixed using https://review.openstack.org/#/c/140121/

whereas if you have any other region applicable you can specify the same in configs.

Mark Biciunas (mbiciunas) wrote :

Thank you for responding.

However, os_region_name was specified in trove-taskmanager.conf and is a valid region. So this can't be the issue.

Also, there is only one region defined = RegionOne.

The following is the relevant section from trove-taskmanager.conf:

    # Configuration options for talking to nova via the novaclient.
    trove_auth_url = http://10.40.10.230:5000/v2.0
    #nova_compute_url = http://localhost:8774/v2
    #nova_compute_url = http://10.40.10.230:8774/v2
    #cinder_url = http://localhost:8776/v1
    #cinder_url = http://10.40.10.230:8776/v1
    #swift_url = http://localhost:8080/v1/AUTH_
    #swift_url = http://10.40.10.230:8080/v1/AUTH_
    #neutron_url = http://localhost:9696/

    # nova_compute_url, cinder_url, swift_url, and heat_url can all be fetched
    # from Keystone. To fetch from Keystone, comment out nova_compute_url,
    # cinder_url, swift_url, and heat_url and optionally uncomment the lines below.

    # Region name of this node. Used when searching catalog. Default value is None.
    os_region_name = RegionOne
    # Service type to use when searching catalog.
    nova_compute_service_type = compute
    # Service type to use when searching catalog.
    cinder_service_type = volumev2
    # Service type to use when searching catalog.
    swift_service_type = object-store
    # Service type to use when searching catalog.
    #heat_service_type = orchestration
    # Service type to use when searching catalog.
    #neutron_service_type = network

Some notes about our configuration:
    - Network is nova-network, no neutron.
    - Distribution is RedHat Openstack 5 running on Red Hat Linux 7
    - Installed using packstack
    - All services are on one machine, save for Trove which has been installed to a separate machine.

Finally, as per ther orginal bug filing, Trove works if the urls are specified. So the issue is strictly with url lookup.

I think the status on this bug should be changed back to New and properly investigated. If there is further information I can give you to help reproduce, please let me know.

Changed in trove:
status: Invalid → New
Changed in trove:
importance: Undecided → Medium
assignee: nobody → Nikhil Manchanda (slicknik)
Javier Peña (jpena-c) wrote :

I was hit by the same issue. As a workaround, setting "exists_notification_transformer = " (empty value) in trove-taskmanager.conf worked for me.

I'm quite sure I got the solution somewhere online, but I cannot find it now, so leaving it here in case it helps someone else.

Changed in puppet-trove:
status: New → Triaged
Doug Shelley (0-doug) wrote :

The issue here is that the Task Manager sets up an exists notification transformer during it's startup sequence:

(from taskmanager/manager.py)
     self.admin_context = TroveContext(
            user=CONF.nova_proxy_admin_user,
            auth_token=CONF.nova_proxy_admin_pass,
            tenant=CONF.nova_proxy_admin_tenant_id)
        if CONF.exists_notification_transformer:
            self.exists_transformer = importutils.import_object(
                CONF.exists_notification_transformer,
                context=self.admin_context)

The admin_context hasn't gone through the process of authenticating so the service_catalog property isn't set. During the initialization of the NovaNotificationTransformer:
(from extensions/mgmt/instances/models.py)
class NovaNotificationTransformer(NotificationTransformer):
    def __init__(self, **kwargs):
        super(NovaNotificationTransformer, self).__init__(**kwargs)
        self.context = kwargs['context']
        self.nova_client = remote.create_admin_nova_client(self.context)
        self._flavor_cache = {}

If you don't set CONF.nova_compute_url, the call to create_admin_nova_client will attempt to use the context.service_catalog which in this case is never set.

In order for this code to work, I believe the Task Manager is going to have to authenticate the admin_context with Keystone to get the service_catalog.

(BTW, the workaround mentioned above works because this code path is not executed if exists_notification_transformer is None.)

Amrith Kumar (amrith) on 2015-10-21
Changed in trove:
status: New → Confirmed
Amrith Kumar (amrith) wrote :

Nikhil, any word on this?

Changed in trove:
milestone: none → newton-1
Amrith Kumar (amrith) on 2016-09-18
Changed in trove:
milestone: newton-1 → ongoing
Changed in trove:
assignee: Nikhil Manchanda (slicknik) → Shaik Apsar (sa709c)
status: Confirmed → In Progress
Shaik Apsar (sa709c) wrote :

I'm done with the changes for the review and please review the changes.

Now trove-taskmanager service is able to startup without nova_compute_url and we don't need to be disable exists_notification_transformer as workaround.

Since we are using nova admin credentials with valid trove_auth_url, 'trove.extensions.mgmt.instances.models.NovaNotificationTransformer' can make use of trove.common.single_tenant_remote module to get the nova admin client.

Chhavi Agarwal (chhagarw) wrote :

Can you share the review for this bug, so I can have a look, I am also facing the similar issue

Chhavi Agarwal (chhagarw) wrote :

Checked the proposed fix, but this is not fixing the actual issue for EmptyCatalogue, rather its skipping the service catalogue, and using the single_remote_tenant.
To actually fix this issue, we need to use the Keystone Session while creating the TroveContext, and pass the actual auth_token, this will initialize the service_catalogue for Trove.
Similar approach is been used in openstack-heat project
https://github.com/openstack/heat/blob/master/heat/engine/clients/os/nova.py#L81

Change abandoned by amrith (<email address hidden>) on branch: master
Review: https://review.openstack.org/385584
Reason: abandoning for inactivity

Changed in trove:
assignee: Shaik Apsar (sa709c) → Chhavi Agarwal (chhagarw)
Manoj Kumar (manojnkumar) wrote :

Chhavi, do you plan to drop the change you proposed?

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers