create volume in scaling policy fails

[Mike Spreitzer]

Create a stack from a template that includes (a) an OS::Heat::AutoScalingGroup whose member is a stack that contains, perhaps among other resources, an OS::Cinder::Volume, and (b) an OS::Heat::ScalingPolicy for scaling that ASG up by 1 member. Observe that creation completes successfully. The do an HTTP POST to the webhook URL of the scaling policy. You will get a failure; the following excerpt from the heat engine's log illustrates the pathology:

2014-06-22 16:17:20.149 25611 ERROR heat.engine.resource [req-95c63832-5561-4665-8c8d-fdf22b41b7a7 None] CREATE : CinderVolume "my_vol" Stack "T7-asg-6feqp2fsbmn2-4fjlpundvdc2-buquivhkqec2" [ed73bebd-7175-4a7b-ac64-4f9ab983e656]
2014-06-22 16:17:20.149 25611 TRACE heat.engine.resource Traceback (most recent call last):
2014-06-22 16:17:20.149 25611 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/resource.py", line 395, in _do_action
2014-06-22 16:17:20.149 25611 TRACE heat.engine.resource handle())
2014-06-22 16:17:20.149 25611 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/resources/volume.py", line 110, in handle_create
2014-06-22 16:17:20.149 25611 TRACE heat.engine.resource cinder = self.cinder()
2014-06-22 16:17:20.149 25611 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/resource.py", line 351, in cinder
2014-06-22 16:17:20.149 25611 TRACE heat.engine.resource return self.stack.clients.cinder()
2014-06-22 16:17:20.149 25611 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/clients.py", line 187, in cinder
2014-06-22 16:17:20.149 25611 TRACE heat.engine.resource self._clients['cinder'] = cinderclient.Client('1', **args)
2014-06-22 16:17:20.149 25611 TRACE heat.engine.resource File "/opt/stack/python-cinderclient/cinderclient/client.py", line 571, in Client
2014-06-22 16:17:20.149 25611 TRACE heat.engine.resource return client_class(*args, **kwargs)
2014-06-22 16:17:20.149 25611 TRACE heat.engine.resource File "/opt/stack/python-cinderclient/cinderclient/v1/client.py", line 103, in __init__
2014-06-22 16:17:20.149 25611 TRACE heat.engine.resource session=session)
2014-06-22 16:17:20.149 25611 TRACE heat.engine.resource File "/opt/stack/python-cinderclient/cinderclient/client.py", line 550, in _construct_http_client
2014-06-22 16:17:20.149 25611 TRACE heat.engine.resource auth_plugin=auth_plugin,
2014-06-22 16:17:20.149 25611 TRACE heat.engine.resource File "/opt/stack/python-cinderclient/cinderclient/client.py", line 187, in __init__
2014-06-22 16:17:20.149 25611 TRACE heat.engine.resource self.auth_url = auth_url.rstrip('/')
2014-06-22 16:17:20.149 25611 TRACE heat.engine.resource AttributeError: 'NoneType' object has no attribute 'rstrip'

Indeed, self.context has auth_url=None at line 395 in heat/engine/resource.py. That's the way it works for scaling policies. For comparison, see the code for OS::Nova::Server; it correctly copes with this. Here is the to_dict() of self.context at that point:

{'username': None, 'user_id': None, 'show_deleted': False, 'roles': [], 'tenant_id': u'77c2f7a9eedb423fac35af5d38c40e04',
'auth_token': None, 'auth_token_info': None, 'auth_url': None, 'trust_id': u'76dee940b2c44fe98577269490bc12c1',
'request_id': u'req-95c63832-5561-4665-8c8d-fdf22b41b7a7', 'is_admin': False,
'trustor_user_id': u'0d0ed13ccab944f8a9ad72bc652a692b', 'password': None, 'aws_creds': None,
'tenant': u'admin', 'user': None}

[Mike Spreitzer (mike-spreitzer)]

I tested with the asg_of_stacks.yaml of https://review.openstack.org/#/c/97366

[Mike Spreitzer (mike-spreitzer)]

FYI, here the debug logging from the point where an auth token is requested from the trust; note that the returned token now uses PKIZ (I think there was a recent change from PKI):

2014-06-22 16:17:18.670 25611 DEBUG keystoneclient.session [req-95c63832-5561-4665-8c8d-fdf22b41b7a7 ] REQ: curl -i -X POST http://10.10.0.123:5000/v3/auth/tokens -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python-keystoneclient" -d '{"auth": {"scope": {"OS-TRUST:trust": {"id": "76dee940b2c44fe98577269490bc12c1"}}, "identity": {"password": {"user": {"domain": {"id": "default"}, "password": "DevStakNow", "name": "heat"}}, "methods": ["password"]}}}' request /opt/stack/python-keystoneclient/keystoneclient/session.py:252
2014-06-22 16:17:18.671 25611 INFO urllib3.connectionpool [req-95c63832-5561-4665-8c8d-fdf22b41b7a7 ] Starting new HTTP connection (1): 10.10.0.123
2014-06-22 16:17:18.672 25611 DEBUG urllib3.connectionpool [req-95c63832-5561-4665-8c8d-fdf22b41b7a7 ] Setting read timeout to None _make_request /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:375
2014-06-22 16:17:18.849 25611 DEBUG urllib3.connectionpool [req-95c63832-5561-4665-8c8d-fdf22b41b7a7 ] "POST /v3/auth/tokens HTTP/1.1" 201 5874 _make_request /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:415
2014-06-22 16:17:18.850 25611 DEBUG keystoneclient.session [req-95c63832-5561-4665-8c8d-fdf22b41b7a7 ] RESP: [201] CaseInsensitiveDict({'x-subject-token': 'PKIZ_eJy9Wlt3okoafa9fMe9nndVcNN0-zAMKEohVNAbBqreICVCAsU-MCL9-dhmTzsWkTWbW9FquXFrhu-xv7_0V-ftv_Bs6rsf-NaKX6oe_CfU8vmm4f3ErvHybMitMR7f43S97NHIvwpEVOjs7sibDrMryMhuKkDpW5jRZno6HWfmL5GXhDhptOGzcLMsXxtDzkh-ru2LpDv4RSa8IiuF3rxxEk3h6GcdsFqym7XLONE_eZtetvyaL-bBKV2y7qMfaVTK4V_-xPPd1cdlk3uHNIolv8LVKC-_MO78rrpIQF7YKJqcFoR01AzsvaWd1zI1LKvOSJbMdi8pGJNOC2b4UkVXcqAvWValu4MnUEG5cBVHYEC59yexpLaKyx6LUpJL3mV02vKY7ipdX_I5kYfpvIiEqFOqGuFu4owaTPPH66oKB6_R5Epq0xg3krOUR79PaK27m2p038jdiPr1dGNOOqJCS1m_4nHVLY9CKkb-cjKwirQd3Yk5VEReqWGKEu468Vti0L-pQo3Vo8s7RSBCNc3UnIXkvsP2CueOSRZbGW0_d6Z4nerXPu84rPp_epOY058bmZmEu78XcK8hNvA-puj4frtM67ubGWHsolNfQKNxQm2-o9Gc0Ss8eXtk96zKN2mE_Kbw7gg83CPeXSJ56Wy4MPb96xMBj5etpJeqxvjgPC5Ui2r4Rl96ZuoAjEpEvk53mrfbRPOZrMoO2Qi7zIIkr4XoalazmCe1YMq5ZEnaBSzVC3VnDRq_zHZeLpLo_XFAi93zhDlb7gpt3hUj6ZeoO1ovVYxfwy-XcvzsUS1ueW2eT9seOjnrq1Uyk09Lutse61JiYfDfpUnTAr5nh9AlgWnIZttQQOe-ozhOuo1U93gEb6MYDkPxKGNV2oQA08i9FslwvjP7PRT0riCrIVdLXxNxHfuyQQnW_dOMWhbrbX-AJws6O2WkTRJnGuriiMjSBgwq9FzkqXAUJNQDfPjO83T7_8z-nRt7L7dTUyHu5HU_t0J1naaELaoQy5JUa1BYY62nFpNOndqYB4gZ1xzUGTQfyjqZFPtOyY2mRz7TsWFrEWw11Xu_Wh-l7hHEvSHiLKHTVMhF5GsMHuBznLPF2vCtNLlOdgkMIG_mD-QP_9VN3tue6JbhgqWD9AgOeISRGPWGlsDlmJGyDaJkjhVxSlxo8CjtWg3lqrj98sJ8vkocLcqO6Bwe2j0X86iz0qSzb13UiX-0_2m8yl7aESdC5pC2LVM_9QshhIWoOTPAd62Y7KqvqI1iTU8L8KEry5wnMc4rJY_VY8i7tGDBD7VkPncDN0pYEdoWxRH_l9MNQ34uUfBZ5r4FHHpDnF0HidNROd6KeadwINWZ7bQDqZrVfQ5F20M2cGX4pLl8Cby9tV4meC-Pxjg_KS7uwx2zHFLaj8W6GC_qKmTGVmR64XBM17YF-OoJvXvA8olkt3PE9xwWVIkMnqkU9bRZA4_L8Ser0tG72NyJX51MttW-3k85RBcLLuqeR1wUSwt89XByKtXr0Dl51qFk5uMcN98qkMF2lNWbcOGA_mW6UMj1PCSTTA7F0ovZLZmeoSbjjqBvhUYUegxwiVvJkXEH_HkN_CvOjKMkpYR6L8tDeeyCxeRXqbBfYMx0q-PDVZhW0EJQ_U6zV4xIyYHh9mvB9pOTUUN-LlPwp1PQ8LhZuJV_MBwrJZaYDAlAmV3HdNBdRXHC5LCH1BcxOH...

[Pavlo Shchelokovskyy (pshchelo)]

I think this bug is related to one I've filed for Horizon
https://bugs.launchpad.net/horizon/+bug/1331406

related Keystone commit
21bf6c7fdae14a5e2143c3bad4914e30c982f5d4 (Default to PKIZ tokens)

[Mike Spreitzer (mike-spreitzer)]

One workaround, maybe even fix, is to add a little code to the __init__ method of the HTTPClient class of python-cinderclient/cinderclient/client.py. Before the first use of auth_url, add this code:

      if auth_url is None:
          auth_url = ''

[William C. Arnold (barnold-8)]

This is related to https://bugs.launchpad.net/heat/+bug/1334492 "Delete volume attachment ine in scaling policy fails" [my typo in defect]

There, i noticed that some python clients fix the the root cause:
Some of clients e.g. python-neutronclient and python-novaclient do the following
        self.auth_url = auth_url.rstrip('/') if auth_url else auth_url

and some fail if passed None for auth_url, e.g. python-cinderclient and python-troveclient have
        self.auth_url = auth_url.rstrip('/')

I guess these is a client problem unless heat wants a workaround?

[Zane Bitter (zaneb)]

If the patch in the comments of bug 1334492 fixes this too, then let's get it in ASAP.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/124481

[Mike Spreitzer (mike-spreitzer)]

FYI: following on comment #5 here, Bill opened https://bugs.launchpad.net/python-cinderclient/+bug/1358926 against python-cinderclient. That one also has a fix in review.

[Mike Spreitzer (mike-spreitzer)]

For the sake of completeness of the record here: this bug and https://bugs.launchpad.net/heat/+bug/1334492 are two symptoms of the same problem, which could also be considered to be a problem in python-cinderclient --- where it is known as https://bugs.launchpad.net/python-cinderclient/+bug/1358926

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/124481
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=8d4b3ef9ac7ff5d52ae9a81d22ba1d1c07b6e749
Submitter: Jenkins
Branch: master

commit 8d4b3ef9ac7ff5d52ae9a81d22ba1d1c07b6e749
Author: Zane Bitter <email address hidden>
Date: Fri Sep 26 13:57:26 2014 -0400

    Handle clients that don't accept auth_url=None

    Some clients fail when passed None as the auth_url. Pass them an empty
    string instead when the auth_url is None.

    Change-Id: Ie7cf788a00660efae578b6444ee231876025ebba
    Closes-Bug: #1332991
    Closes-Bug: #1334492