Delete volume attachment ine in scaling policy fails

[William C. Arnold]

This is related to https://bugs.launchpad.net/heat/+bug/1332991, which is a failure on scale-up.
Create a stack from a template that includes (a) an OS::Heat::AutoScalingGroup whose member is a stack that contains, perhaps among other resources, an OS::Cinder::Volume, and (b) an OS::Heat::ScalingPolicy for scaling that ASG down by 1 member.
The patch https://review.openstack.org/#/c/97366, specifically hot/asg_of_stacks.yaml and hot/vm_with_cinder.yaml, is an working example.
Observe that stack creation completes successfully. Then do an HTTP POST to the webhook URL of the scale down policy. You will get a failure.
See below. The python-cinderclient appears to be unforgiving of a null auth_url. The failure is when the context is a trust.

2014-06-25 21:32:41.122 17728 TRACE heat.engine.resource Traceback (most recent call last):
2014-06-25 21:32:41.122 17728 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/resource.py", line 671, in delete
2014-06-25 21:32:41.122 17728 TRACE heat.engine.resource handle_data = self.handle_delete()
2014-06-25 21:32:41.122 17728 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/resources/volume.py", line 467, in handle_delete
2014-06-25 21:32:41.122 17728 TRACE heat.engine.resource scheduler.TaskRunner(detach_task)()
2014-06-25 21:32:41.122 17728 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/scheduler.py", line 141, in __call__
2014-06-25 21:32:41.122 17728 TRACE heat.engine.resource self.start(timeout=timeout)
2014-06-25 21:32:41.122 17728 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/scheduler.py", line 162, in start
2014-06-25 21:32:41.122 17728 TRACE heat.engine.resource self.step()
2014-06-25 21:32:41.122 17728 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/scheduler.py", line 190, in step
2014-06-25 21:32:41.122 17728 TRACE heat.engine.resource next(self._runner)
2014-06-25 21:32:41.122 17728 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/resources/volume.py", line 363, in __call__
2014-06-25 21:32:41.122 17728 TRACE heat.engine.resource vol = self.clients.cinder().volumes.get(nova_vol.id)
2014-06-25 21:32:41.122 17728 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/clients.py", line 189, in cinder
2014-06-25 21:32:41.122 17728 TRACE heat.engine.resource self._clients['cinder'] = cinderclient.Client('1', **args)
2014-06-25 21:32:41.122 17728 TRACE heat.engine.resource File "/opt/stack/python-cinderclient/cinderclient/client.py", line 571, in Client
2014-06-25 21:32:41.122 17728 TRACE heat.engine.resource return client_class(*args, **kwargs)
2014-06-25 21:32:41.122 17728 TRACE heat.engine.resource File "/opt/stack/python-cinderclient/cinderclient/v1/client.py", line 103, in __init__
2014-06-25 21:32:41.122 17728 TRACE heat.engine.resource session=session)
2014-06-25 21:32:41.122 17728 TRACE heat.engine.resource File "/opt/stack/python-cinderclient/cinderclient/client.py", line 550, in _construct_http_client
2014-06-25 21:32:41.122 17728 TRACE heat.engine.resource auth_plugin=auth_plugin,
2014-06-25 21:32:41.122 17728 TRACE heat.engine.resource File "/opt/stack/python-cinderclient/cinderclient/client.py", line 187, in __init__
2014-06-25 21:32:41.122 17728 TRACE heat.engine.resource self.auth_url = auth_url.rstrip('/')
2014-06-25 21:32:41.122 17728 TRACE heat.engine.resource AttributeError: 'NoneType' object has no attribute 'rstrip'

Drilling down, this is in python-cinderclient, in cinderclient/client.py, approximately here:
Traceback (most recent call last):
  File "/opt/stack/heat/heat/engine/resources/volume.py", line 366, in __call__
    vol = self.clients.cinder().volumes.get(nova_vol.id)
  File "/opt/stack/heat/heat/engine/clients.py", line 189, in cinder
    self._clients['cinder'] = cinderclient.Client('1', **args)
  File "/opt/stack/python-cinderclient/cinderclient/client.py", line 571, in Client
    return client_class(*args, **kwargs)
  File "/opt/stack/python-cinderclient/cinderclient/v1/client.py", line 103, in __init__
    session=session)
  File "/opt/stack/python-cinderclient/cinderclient/client.py", line 550, in _construct_http_client
    auth_plugin=auth_plugin,
  File "/opt/stack/python-cinderclient/cinderclient/client.py", line 187, in __init__
    self.auth_url = auth_url.rstrip('/')
AttributeError: 'NoneType' object has no attribute 'rstrip'

The following hack resolved the problem, not sure if it is even close to correct but it confirms the problem
--- a/heat/engine/clients.py
+++ b/heat/engine/clients.py
@@ -177,7 +177,7 @@ class OpenStackClients(object):
         endpoint_type = self._get_client_option('cinder', 'endpoint_type')
         args = {
             'service_type': 'volume',
- 'auth_url': con.auth_url,
+ 'auth_url': con.auth_url if con.auth_url is not None else '',
             'project_id': con.tenant,
             'username': None,
             'api_key': None,

[William C. Arnold (barnold-8)]

Looking over the python clients, some of them e.g. python-neutronclient and python-novaclient do the following
        self.auth_url = auth_url.rstrip('/') if auth_url else auth_url

and some fail if passed None for auth_url, e.g. python-cinderclient and python-troveclient have
        self.auth_url = auth_url.rstrip('/')

I guess this is a client problem unless heat wants a workaround?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/124481

[Mike Spreitzer (mike-spreitzer)]

FYI: following on comment #1 here, Bill opened https://bugs.launchpad.net/python-cinderclient/+bug/1358926 against python-cinderclient.

[Mike Spreitzer (mike-spreitzer)]

For the sake of completeness of the record here: this bug and https://bugs.launchpad.net/heat/+bug/1332991 are two symptoms of the same problem, which could also be considered to be a problem in python-cinderclient --- where it is known as https://bugs.launchpad.net/python-cinderclient/+bug/1358926

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/124481
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=8d4b3ef9ac7ff5d52ae9a81d22ba1d1c07b6e749
Submitter: Jenkins
Branch: master

commit 8d4b3ef9ac7ff5d52ae9a81d22ba1d1c07b6e749
Author: Zane Bitter <email address hidden>
Date: Fri Sep 26 13:57:26 2014 -0400

    Handle clients that don't accept auth_url=None

    Some clients fail when passed None as the auth_url. Pass them an empty
    string instead when the auth_url is None.

    Change-Id: Ie7cf788a00660efae578b6444ee231876025ebba
    Closes-Bug: #1332991
    Closes-Bug: #1334492