heatclient resolves paths for types and get_file calls that then don't make sense in swift

[Dougal Matthews]

There is currently a problem with templates that are uploaded to Swift. After heatclient is used to fetch all the templates the paths to files are resolved to absolute paths like file:///home/stack/tripleo-heat-templates/puppet/manifests/overcloud_controller_pacemaker.pp.

This then breaks the deployment. This can be tested like this.:

    openstack overcloud deploy \
    --templates \
    -e ~/tripleo-heat-templates/environments/puppet-pacemaker.yaml \
    -e ~/tripleo-heat-templates/environments/storage-environment.yaml \
    -e ~/tripleo-heat-templates/environments/ceph-radosgw.yaml \
    --ceph-storage-scale 1 \
    --debug;

Either tripleoclient is doing too much, or not enough. I'm not sure. The problem is likely in the logic here: https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/overcloud_deploy.py#L283-L287

[Dougal Matthews (d0ugal)]

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tripleo_common/actions/templates.py", line 188, in run
    object_request=_object_request))
  File "/usr/lib/python2.7/site-packages/heatclient/common/template_utils.py", line 247, in process_multiple_environments_and_files
    include_env_in_files=include_env_in_files)
  File "/usr/lib/python2.7/site-packages/heatclient/common/template_utils.py", line 296, in process_environment_and_files
    env_base_url, is_object=True, object_request=object_request)
  File "/usr/lib/python2.7/site-packages/heatclient/common/template_utils.py", line 346, in resolve_environment_urls
    is_object=is_object, object_request=object_request)
  File "/usr/lib/python2.7/site-packages/heatclient/common/template_utils.py", line 164, in get_file_contents
    object_request=object_request)[1]
  File "/usr/lib/python2.7/site-packages/heatclient/common/template_utils.py", line 99, in get_template_contents
    object_request)
  File "/usr/lib/python2.7/site-packages/heatclient/common/template_utils.py", line 120, in resolve_template_get_files
    ignore_if, recurse_if, is_object, object_request)
  File "/usr/lib/python2.7/site-packages/heatclient/common/template_utils.py", line 144, in get_file_contents
    is_object, object_request)
  File "/usr/lib/python2.7/site-packages/heatclient/common/template_utils.py", line 144, in get_file_contents
    is_object, object_request)
  File "/usr/lib/python2.7/site-packages/heatclient/common/template_utils.py", line 144, in get_file_contents
    is_object, object_request)
  File "/usr/lib/python2.7/site-packages/heatclient/common/template_utils.py", line 144, in get_file_contents
    is_object, object_request)
  File "/usr/lib/python2.7/site-packages/heatclient/common/template_utils.py", line 144, in get_file_contents
    is_object, object_request)
  File "/usr/lib/python2.7/site-packages/heatclient/common/template_utils.py", line 144, in get_file_contents
    is_object, object_request)
  File "/usr/lib/python2.7/site-packages/heatclient/common/template_utils.py", line 144, in get_file_contents
    is_object, object_request)
  File "/usr/lib/python2.7/site-packages/heatclient/common/template_utils.py", line 157, in get_file_contents
    file_content = object_request('GET', str_url)
  File "/usr/lib/python2.7/site-packages/tripleo_common/actions/templates.py", line 178, in _object_request
    method, url, headers={'X-Auth-Token': token}).content
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 57, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 579, in send
    adapter = self.get_adapter(url=request.url)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 653, in get_adapter
    raise InvalidSchema("No connection adapters were found for '%s'" % url)
InvalidSchema: No connection adapters were found for 'file:///home/stack/tripleo-heat-templates/puppet/manifests/overcloud_controller_pacemaker.pp'

[Giulio Fidente (gfidente)]

Initial investigation suggests that template_url is set to template_object in :

  https://github.com/openstack/python-heatclient/blob/master/heatclient/common/template_utils.py#L95

which then goes to :

  https://github.com/openstack/python-heatclient/blob/master/heatclient/common/utils.py#L351

the manipulation seems to be happening in the heatclient

[Steven Hardy (shardy)]

So, testing locally I can't reproduce this, but note there are issues with *updating* a plan, as discussed in https://bugs.launchpad.net/tripleo/+bug/1624727

If you ensure you start without a plan, then I think things work OK, e.g:

openstack overcloud plan delete overcloud

Then I did

$ openstack overcloud deploy --templates /tmp/tripleo-heat-templates -e /tmp/tripleo-heat-templates/environments/puppet-pacemaker.yaml -e /tmp/tripleo-heat-templates/environments/storage-environment.yaml -e /tmp/tripleo-heat-templates/environments/ceph-radosgw.yaml --ceph-storage-scale 1 --debug

$ swift list
opstools
ov-gxjconnlncm-0-lhtpht74rtdt-Controller-3ya6xaqtfxzs
ov-jweawxaryt-0-ehen4o5kpz5o-CephStorage-vmx4ickfofnl
ov-vavghyq4wc-0-pzhv376qi24l-NovaCompute-tonqa3opzpwp
overcloud
overcloud-artifacts

$ heat stack-list
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
+--------------------------------------+------------+--------------------+----------------------+--------------+
| id | stack_name | stack_status | creation_time | updated_time |
+--------------------------------------+------------+--------------------+----------------------+--------------+
| 7a3d0c3e-27bf-46bb-addb-d615c177ab31 | overcloud | CREATE_IN_PROGRESS | 2016-09-19T17:48:00Z | None |
+--------------------------------------+------------+--------------------+----------------------+--------------+

So, I'd ensure you definitely don't have any plan, then re-test?

Also ensure you have all the very latest tripleo-common pieces, and that you either re-run openstack undercloud install or do this to update the actions/workflows in mistral:

$ cat mistral_update.sh
#!/bin/bash
sudo mistral-db-manage populate
sleep 1
sudo systemctl restart openstack-mistral-engine
sudo systemctl restart openstack-mistral-executor
sudo systemctl restart openstack-mistral-api
sleep 1

for workbook in $(mistral workbook-list | grep tripleo | cut -f 2 -d ' '); do
    mistral workbook-delete $workbook
done
for workflow in $(mistral workflow-list | grep tripleo | cut -f 2 -d ' '); do
    mistral workflow-delete $workflow
done
for workbook in $(ls /usr/share/openstack-tripleo-common/workbooks/*); do
    mistral workbook-create $workbook
done

[Steven Hardy (shardy)]

Ok, so update - after some IRC discussions I can now reproduce - the problem is only triggered if you use relative paths on the CLI, using absolute paths for everything seems to be a workaround.

So we're doing something wrong in tripleoclient with relative paths, so presumably the files map is missing the required file (or it has the wrong key), so then the mistral action blows up trying to read local files when processing the templates.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/373220

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/373220
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=0ab4114b1b9d9de18835e44adef8f580bb418306
Submitter: Jenkins
Branch: master

commit 0ab4114b1b9d9de18835e44adef8f580bb418306
Author: Steven Hardy <email address hidden>
Date: Tue Sep 20 12:39:43 2016 +0100

    Always use absolute path for tht_root

    If you don't use the absolute path then the logic in
    _process_and_upload_environment breaks and deployment fails

    We probably need a more comprehensive fix but this provides a
    stopgap solution while we figure out how to rework this.

    Change-Id: I9c6a7394d1f831ab7f0b774f675f77f924fea335
    Partial-Bug: #1623552

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/373487

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/373487
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=3b31ea165d99a24da393b6063d5321620d07025f
Submitter: Jenkins
Branch: stable/newton

commit 3b31ea165d99a24da393b6063d5321620d07025f
Author: Steven Hardy <email address hidden>
Date: Tue Sep 20 12:39:43 2016 +0100

    Always use absolute path for tht_root

    If you don't use the absolute path then the logic in
    _process_and_upload_environment breaks and deployment fails

    We probably need a more comprehensive fix but this provides a
    stopgap solution while we figure out how to rework this.

    Change-Id: I9c6a7394d1f831ab7f0b774f675f77f924fea335
    Partial-Bug: #1623552
    (cherry picked from commit 0ab4114b1b9d9de18835e44adef8f580bb418306)

[Emilien Macchi (emilienm)]

can we close it now?

[Steven Hardy (shardy)]

Ok, so I think we solved one part of the problem here, and there's another part of the problem described in https://bugs.launchpad.net/tripleo/+bug/1624727

So I'll close this one fix committed, and we can follow up on the remaining issues related to local file resolution in https://bugs.launchpad.net/tripleo/+bug/1624727