stack create failed when using multiattach cinder volume

[Linda Yu]

Openstack version: Newton, Nova: 14.0.7 Heat: 7.0.4 Cinder:9.1.4
I defined 2 nova servers, 1 cinder volume, and 2 cinder VolumeAttachment in heat template as below.

    node:0-0-1:0-0-9:shared_vol1:
        type: 'OS::Cinder::Volume'
        properties:
            availability_zone: nova
            size: '5'
            name: 'shared_vol'
            volume_type: LVM
            multiattach: 'True'
    'node:0-0-1:attach':
        type: 'OS::Cinder::VolumeAttachment'
        properties:
            instance_uuid:
                get_resource: 'node:0-0-1:server'
            mountpoint: /dev/vdc
            volume_id:
                get_resource: 'node:0-0-1:0-0-9:shared_vol1'
    'node:0-0-9:attach':
        type: 'OS::Cinder::VolumeAttachment'
        properties:
            instance_uuid:
                get_resource: 'node:0-0-9:server'
           mountpoint: /dev/vdc
            volume_id:
                get_resource: 'node:0-0-1:0-0-9:shared_vol1'
        depends_on: 'node:0-0-1:attach'
    'node:0-0-1:server':
        type: 'OS::Nova::Server'
        properties:
… …
    'node:0-0-9:server':
        type: 'OS::Nova::Server'
        properties:
… …

When run heat stack-create, it failed with following error information:
stack_status_reason | Resource CREATE failed: Unauthorized: | resources.node:0-0-9:attach: The request you have made
| | requires authentication. (HTTP 401)

Below is the heat-engine.log.
2017-07-24 14:33:45.352 5013 INFO heat.engine.resource [req-bbd4a919-c377-432c-a100-95d01b1cc307 fc607bdc62e3489c8d89696831c0906a 5da29d321481424fb9a318f9e52eff49 - - -] creating Server "node:0-0-2:server" Stack "rezhong-test2" [c3210c39-072b-4565-b916-d2beac737892]
2017-07-24 14:33:57.308 5017 INFO heat.engine.resource [req-bbd4a919-c377-432c-a100-95d01b1cc307 fc607bdc62e3489c8d89696831c0906a 5da29d321481424fb9a318f9e52eff49 - - -] creating Server "node:0-0-1:server" Stack "rezhong-test2" [c3210c39-072b-4565-b916-d2beac737892]
2017-07-24 14:34:11.352 5006 INFO heat.engine.resource [req-bbd4a919-c377-432c-a100-95d01b1cc307 fc607bdc62e3489c8d89696831c0906a 5da29d321481424fb9a318f9e52eff49 - - -] creating CinderVolumeAttachment "node:0-0-1:attach" Stack "rezhong-test2" [c3210c39-072b-4565-b916-d2beac737892]
2017-07-24 14:34:15.508 5006 INFO heat.engine.clients.os.cinder [req-bbd4a919-c377-432c-a100-95d01b1cc307 fc607bdc62e3489c8d89696831c0906a 5da29d321481424fb9a318f9e52eff49 - - -] Attaching volume 0421f33f-ce4c-4c9f-904a-d8f2d60bfbf3 complete
2017-07-24 14:34:15.679 5018 INFO heat.engine.resource [req-bbd4a919-c377-432c-a100-95d01b1cc307 fc607bdc62e3489c8d89696831c0906a 5da29d321481424fb9a318f9e52eff49 - - -] creating CinderVolumeAttachment "node:0-0-9:attach" Stack "rezhong-test2" [c3210c39-072b-4565-b916-d2beac737892]
2017-07-24 15:33:38.228 5018 INFO heat.engine.resource [req-bbd4a919-c377-432c-a100-95d01b1cc307 fc607bdc62e3489c8d89696831c0906a 5da29d321481424fb9a318f9e52eff49 - - -] CREATE: CinderVolumeAttachment "node:0-0-9:attach" [0421f33f-ce4c-4c9f-904a-d8f2d60bfbf3] Stack "rezhong-test2" [c3210c39-072b-4565-b916-d2beac737892]
2017-07-24 15:33:38.228 5018 ERROR heat.engine.resource Traceback (most recent call last):
2017-07-24 15:33:38.228 5018 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 763, in _action_recorder
2017-07-24 15:33:38.228 5018 ERROR heat.engine.resource yield
2017-07-24 15:33:38.228 5018 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 865, in _do_action
2017-07-24 15:33:38.228 5018 ERROR heat.engine.resource yield self.action_handler_task(action, args=handler_args)
2017-07-24 15:33:38.228 5018 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 353, in wrapper
2017-07-24 15:33:38.228 5018 ERROR heat.engine.resource step = next(subtask)
2017-07-24 15:33:38.228 5018 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 816, in action_handler_task
2017-07-24 15:33:38.228 5018 ERROR heat.engine.resource done = check(handler_data)
2017-07-24 15:33:38.228 5018 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/volume_base.py", line 192, in check_create_complete
2017-07-24 15:33:38.228 5018 ERROR heat.engine.resource return self.client_plugin().check_attach_volume_complete(volume_id)
2017-07-24 15:33:38.228 5018 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/clients/os/cinder.py", line 157, in check_attach_volume_complete
2017-07-24 15:33:38.228 5018 ERROR heat.engine.resource vol = self.client().volumes.get(vol_id)
2017-07-24 15:33:38.228 5018 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/cinderclient/v3/volumes.py", line 289, in get
2017-07-24 15:33:38.228 5018 ERROR heat.engine.resource return self._get("/volumes/%s" % volume_id, "volume")
2017-07-24 15:33:38.228 5018 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/cinderclient/base.py", line 308, in _get
2017-07-24 15:33:38.228 5018 ERROR heat.engine.resource resp, body = self.api.client.get(url)
2017-07-24 15:33:38.228 5018 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 143, in get
2017-07-24 15:33:38.228 5018 ERROR heat.engine.resource return self._cs_request(url, 'GET', **kwargs)
2017-07-24 15:33:38.228 5018 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 134, in _cs_request
2017-07-24 15:33:38.228 5018 ERROR heat.engine.resource return self.request(url, method, **kwargs)
2017-07-24 15:33:38.228 5018 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 123, in request
2017-07-24 15:33:38.228 5018 ERROR heat.engine.resource raise exceptions.from_response(resp, body)
2017-07-24 15:33:38.228 5018 ERROR heat.engine.resource Unauthorized: The request you have made requires authentication. (HTTP 401)
2017-07-24 15:33:38.228 5018 ERROR heat.engine.resource
2017-07-24 15:33:38.335 5018 INFO heat.engine.stack [req-bbd4a919-c377-432c-a100-95d01b1cc307 fc607bdc62e3489c8d89696831c0906a 5da29d321481424fb9a318f9e52eff49 - - -] Stack CREATE FAILED (rezhong-test2): Resource CREATE failed: Unauthorized: resources.node:0-0-9:attach: The request you have made requires authentication. (HTTP 401)

[Pavlukhin Max (mpavlukhin)]

Hello, Linda Yu!
I found several solutions to solve your problem:

1) Are you sure that you don't forget to get credentials? Did you execute openrc script? Did you execute it with 'admin' argument?

2) Are you sure that you don't forget run stack.sh script after every OS reboot? Try to do the action. You should try to run the script after executing unstack.sh and clean.sh.

Does it help? No? Then we should check that your heat template is correct. Could you give me full version of your template and specify which git branch was used? I tried various templates in stable/ocata. I tried templates that could cause to an error but I had nice results (completely created stacks).

Here example of my template:
https://drive.google.com/file/d/0B_AMW0o7SMLja3pRay14Z2M4WEk/view?usp=sharing

Clean version of this template (edit '<some_statement>'):
https://drive.google.com/file/d/0B_AMW0o7SMLjOFRlRENGcHpCVEk/view?usp=sharing

Thank you for attention!

[Rico Lin (rico-lin)]

Hi Linda could you please certify what Pavlukhin suggest suit for your cases or not?:)