cloud_backend setting results in "maximum recursion depth exceeded"

[Jason Dunsmore]

cloud_backend setting of "heat.engine.clients.Clients" results in "maximum recursion depth exceeded"

To reproduce, set "cloud_backend=heat.engine.clients.Clients" in heat.conf and create a stack.

The user will see:

"ERROR: maximum recursion depth exceeded"

The operator will see a huge traceback in the engine log:

2014-02-11 20:25:20.632 22371 ERROR heat.openstack.common.rpc.amqp [req-b24022ec
-5b8f-481c-a78b-6bf2298a8cd6 None] Exception during message handling
2014-02-11 20:25:20.632 22371 TRACE heat.openstack.common.rpc.amqp Traceback (mo
st recent call last):
2014-02-11 20:25:20.632 22371 TRACE heat.openstack.common.rpc.amqp File "/opt/
stack/heat/heat/openstack/common/rpc/amqp.py", line 462, in _process_data
2014-02-11 20:25:20.632 22371 TRACE heat.openstack.common.rpc.amqp **args)
2014-02-11 20:25:20.632 22371 TRACE heat.openstack.common.rpc.amqp File "/opt/
stack/heat/heat/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2014-02-11 20:25:20.632 22371 TRACE heat.openstack.common.rpc.amqp result =
getattr(proxyobj, method)(ctxt, **kwargs)
2014-02-11 20:25:20.632 22371 TRACE heat.openstack.common.rpc.amqp File "/opt/stack/heat/heat/engine/service.py", line 61, in wrapped
2014-02-11 20:25:20.632 22371 TRACE heat.openstack.common.rpc.amqp return func(self, ctx, *args, **kwargs)
2014-02-11 20:25:20.632 22371 TRACE heat.openstack.common.rpc.amqp File "/opt/stack/heat/heat/engine/service.py", line 389, in create_stack
2014-02-11 20:25:20.632 22371 TRACE heat.openstack.common.rpc.amqp env, **common_params)
2014-02-11 20:25:20.632 22371 TRACE heat.openstack.common.rpc.amqp File "/opt/stack/heat/heat/engine/parser.py", line 86, in __init__
2014-02-11 20:25:20.632 22371 TRACE heat.openstack.common.rpc.amqp self.clients = Clients(context)
2014-02-11 20:25:20.632 22371 TRACE heat.openstack.common.rpc.amqp File "/opt/stack/heat/heat/engine/clients.py", line 339, in __new__
2014-02-11 20:25:20.632 22371 TRACE heat.openstack.common.rpc.amqp context
2014-02-11 20:25:20.632 22371 TRACE heat.openstack.common.rpc.amqp File "/opt/stack/heat/heat/openstack/common/importutils.py", line 38, in import_object
2014-02-11 20:25:20.632 22371 TRACE heat.openstack.common.rpc.amqp return import_class(import_str)(*args, **kwargs)
2014-02-11 20:25:20.632 22371 TRACE heat.openstack.common.rpc.amqp File "/opt/stack/heat/heat/engine/clients.py", line 339, in __new__
2014-02-11 20:25:20.632 22371 TRACE heat.openstack.common.rpc.amqp context
2014-02-11 20:25:20.632 22371 TRACE heat.openstack.common.rpc.amqp File "/opt/stack/heat/heat/openstack/common/importutils.py", line 38, in import_object
2014-02-11 20:25:20.632 22371 TRACE heat.openstack.common.rpc.amqp return import_class(import_str)(*args, **kwargs)
2014-02-11 20:25:20.632 22371 TRACE heat.openstack.common.rpc.amqp File "/opt/stack/heat/heat/engine/clients.py", line 339, in __new__
2014-02-11 20:25:20.632 22371 TRACE heat.openstack.common.rpc.amqp context
2014-02-11 20:25:20.632 22371 TRACE heat.openstack.common.rpc.amqp File "/opt/stack/heat/heat/openstack/common/importutils.py", line 38, in import_object
2014-02-11 20:25:20.632 22371 TRACE heat.openstack.common.rpc.amqp return import_class(import_str)(*args, **kwargs)
2014-02-11 20:25:20.632 22371 TRACE heat.openstack.common.rpc.amqp File "/opt/stack/heat/heat/engine/clients.py", line 339, in __new__
2014-02-11 20:25:20.632 22371 TRACE heat.openstack.common.rpc.amqp context
2014-02-11 20:25:20.632 22371 TRACE heat.openstack.common.rpc.amqp File "/opt/stack/heat/heat/openstack/common/importutils.py", line 38, in import_object
2014-02-11 20:25:20.632 22371 TRACE heat.openstack.common.rpc.amqp return import_class(import_str)(*args, **kwargs)
2014-02-11 20:25:20.632 22371 TRACE heat.openstack.common.rpc.amqp File "/opt/stack/heat/heat/engine/clients.py", line 339, in __new__
[...]

[Steven Hardy (shardy)]

I don't really understand - "heat.engine.clients.Clients" is not a sane setting, what is the use-case here?

Is the bug that we don't catch this corner case and raise an error saying "your config is wrong"?

Or do you think setting heat.engine.clients.Clients should be an alias for heat.engine.clients.OpenStackClients?

[Jason Dunsmore (jasondunsmore)]

I encountered this right after the change that introduced cloud_backend was merged and I was trying various settings (it turned out my plugin_dirs config option needed to be corrected first).

The default is "heat.engine.clients.OpenStackClients", so "heat.engine.clients.Clients" isn't such an insane setting (unless maybe are familiar with the code).

The bug is that the user sees "maximum recursion depth exceeded".

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/144989

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/144989
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=28d9ba7de4cd90857eb647d88183b5e626ed7280
Submitter: Jenkins
Branch: master

commit 28d9ba7de4cd90857eb647d88183b5e626ed7280
Author: tengqm <email address hidden>
Date: Mon Jan 5 22:19:48 2015 +0800

    Deal with misconfigured cloud_backend

    In some rare cases, users may misconfigure 'cloud_backend' in heat.conf.
    The result is that Heat cannot properly import the configured
    module/object. This patch makes Heat throw exceptions that inform users
    of correct error messages and also log errors for operators.

    Closes-Bug: 1279058
    Change-Id: Ied0099f545b0af484aefc50f3d5932111ca3fd0b