Could not fetch contents for file:///home/stack/tripleo-heat-templates/puppet/post.yaml

[Michele Baldessari]

While testing upgrades from M->N, right after I upgraded my undercloud without issues, I get this
when trying to do the first step of the overcloud ugprade (the one that sets the repos).
+ openstack overcloud deploy --templates /home/stack/tripleo-heat-templates --libvirt-type qemu --control-flavor oooq_control --compute-flavor oooq_compute --ceph-storage-flavor oooq_ceph --timeout 75 --ntp-server clock.redhat.com --control-scale 3 --neutron-network-type vxlan --neutron-tunnel-types vxlan -e /home/stack/tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /home/stack/tripleo-heat-templates/environments/network-isolation.yaml -e /home/stack/tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e /home/stack/network-environment.yaml -e /home/stack/tripleo-heat-templates/environments/puppet-pacemaker.yaml -e /home/stack/tripleo-heat-templates/environments/major-upgrade-pacemaker-init.yaml -e /home/stack/overcloud-repo.yaml --ntp-server clock.redhat.com -e /tmp/deploy_env.yaml
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
WARNING: openstackclient.common.exceptions is deprecated and will be removed after Jun 2017. Please use osc_lib.exceptions
Removing the current plan files
Uploading new plan files
Plan updated
Deploying templates in the directory /home/stack/tripleo-heat-templates
Could not fetch contents for file:///home/stack/tripleo-heat-templates/puppet/post.yaml

Both Tom Barron and I are hitting this. I have tried to apply the following reviews before that step:
https://review.openstack.org/#/c/371347/
https://review.openstack.org/#/c/371027
https://review.openstack.org/#/c/368760 (this one is merged but is not yet in the newton rdo repos in current-ci-repo so I had to apply it manually)

The file puppet/post.yaml does not exist. Enly post.j2.yaml

[Tom Barron (tpb)]

Thanks for trying to apply the above reviews, Michele. I hit the issue as follows:

http://paste.fedoraproject.org/429357/14741233/

I can hit this (1) on the first deploy in a brand new environment, (2) redeploy - where I see messages about 'Removing current plan files' and 'Uploading new plan files', and (3) redeploy after first removing the overcloud plan with 'openstack overcloud plan delete overcloud'

This is on master of THT and puppet/tripleo on fresh instack and baremetal vms deployed via tripleo.sh as of about 12 hours ago.

[Dougal Matthews (d0ugal)]

I believe this is essentially a duplicate of https://bugs.launchpad.net/tripleo/+bug/1622683

[Michele Baldessari (michele)]

Thanks for looking at this Dougal. Do we need other reviews than these to get things working:
https://review.openstack.org/#/c/371347/
https://review.openstack.org/#/c/371027
https://review.openstack.org/#/c/368760

I ask because I tried and made sure I had those applied but had no success

[Steven Hardy (shardy)]

@michele - did you re-run openstack undercloud install after applying those patches?

That is required to reload the mistral actions/workflows from the patches touching tripleo-common, or you can do it manually like:

$ cat mistral_update.sh
#!/bin/bash
sudo mistral-db-manage populate
sleep 1
sudo systemctl restart openstack-mistral-engine
sudo systemctl restart openstack-mistral-executor
sudo systemctl restart openstack-mistral-api
sleep 1

for workbook in $(mistral workbook-list | grep tripleo | cut -f 2 -d ' '); do
    mistral workbook-delete $workbook
done
for workflow in $(mistral workflow-list | grep tripleo | cut -f 2 -d ' '); do
    mistral workflow-delete $workflow
done
for workbook in $(ls /usr/share/openstack-tripleo-common/workbooks/*); do
    mistral workbook-create $workbook
done

[Michele Baldessari (michele)]

Thanks Steve. No I had only done the db-manage + service restart without the deletion.

If I do the suggested steps I get:
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
Removing the current plan files
Uploading new plan files
Exception updating plan: The environment is not a valid YAML mapping data type.

Trying to debug it now.

[Tom Barron (tpb)]

I blew away instack and baremetal vms today, yum updated, and built everything fresh. Deployed overcloud as follows:
[stack@instack ~]$ cat > ~/netapp.yaml << _EOF_
> parameter_defaults:
> ManilaNetappLogin: 'snapmirror'
> ManilaNetappPassword: 'ssssshhhhhh'
> ManilaNetappServerHostname: '111.111.111.111'
> ManilaNetappVserver: 'Vserver1'
> ManilaNetappRootVolumeAggr: 'Aggr0'
> ManilaNetappTraceFlags: 'false'
> _EOF_
[stack@instack ~]$ export THT_ROOT=/usr/share/openstack-tripleo-heat-templates
[stack@instack ~]$ openstack overcloud deploy --templates $THT_ROOT \
> -e $THT_ROOT/overcloud-resource-registry-puppet.yaml \
> -e $THT_ROOT/environments/puppet-pacemaker.yaml \
> --control-scale 3 --compute-scale 1\
> --libvirt-type qemu --ntp-server 0.fedora.pool.ntp.org\
> -e $THT_ROOT/environments/manila-netapp-config.yaml \
> -e ./netapp.yaml
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
Creating Swift container to store the plan
Creating plan from template files in: /usr/share/openstack-tripleo-heat-templates
Plan created
Deploying templates in the directory /usr/share/openstack-tripleo-heat-templates
Could not fetch contents for file:///usr/share/openstack-tripleo-heat-templates/puppet/post.yaml

This is not a re-deploy and I'm not using relative paths.

[Tom Barron (tpb)]

Rebuilt everything today and updated all packages but no joy.

http://paste.fedoraproject.org/432017/46694514/

[Ryan Hallisey (rthall14)]

The jinja2 rendering is working properly now. The issue is that heat is not finding the jinja2 files that get rendered in swift in the working THT directory. I don't know what the intended workflow is here: whether to render the templates and have heat pull the rendered files from swift or mistral downloading them to the working THT dir.

Workaround:

move into your active THT dir and run: swift download overcloud

[Tom Barron (tpb)]

> Workaround: move into your active THT dir and run: swift download overcloud

That workaround, in combination with 'openstack overcloud plan delete overcloud', is getting me past this blocker. Thanks!

[Steven Hardy (shardy)]

Ok, thanks for the feedback, definitely sounds like we have some issues due to the current workflow ordering which is:

1. Create plan
2. Plan renders j2 templates (stored in swift)
3. Tripleoclient attempts to locally resolve the files (the pre-rendered tree not the rendered one in swift).

The workaround above seems reasonable, for now, but we need this to be sorted out in tripleoclient really, because it's always going to be possible to reference environment files etc outside of the --templates directory.

[Tom Barron (tpb)]

Oh, and for the record, I'm using:
python-tripleoclient.noarch 5.1.1-0.20160920215837.fdbb7be.el7.centos

[Michele Baldessari (michele)]

I also would like to confirm that the workaround of doing "swift download overcloud" in the tht folder fixes this specific issue for me.

[Emilien Macchi (emilienm)]

Note that it's currently breaking upgrades from Mitaka:
http://logs.openstack.org/10/377910/1/check/gate-tripleo-ci-centos-7-nonha-multinode-updates-nv/edf2bd8/console.html#_2016-09-27_18_58_23_114650

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/379547

[Tom Barron (tpb)]

I did a fresh tripleo.sh install today, new packages and new instack VM, and now see a similar but different failure at overcloud deploy:

 "Error: The following files were not found: /home/stack/tripleo-heat-templates/overcloud-resource-registry-puppet.yaml"

More details at: http://paste.fedoraproject.org/439532/47532303/

The same 'swift download overcloud' into THT workaround appears to work.

[mathieu bultel (mat-bultel)]

Yes, the latest update of the review ^ should handle this issue also.
I put the review as WIP since there is a bunch of things to do before trying to merged it, but you can test it if you want.
I tested it on an upgrade (Mitaka to newton), looks goods for now.
I kicked the gate... lets see how it goes.

[Carlos Camacho (ccamacho)]

Just checking this from last week.

If using env files and if the plan is not created you will find issues. If a prev plan is created it will be "re created"

https://paste.fedoraproject.org/439078/14752428/

[Tom Barron (tpb)]

Just noting that although the "swift download overcloud" into T-H-T workaround "works", it modifies other files than just what is needed. In particular, I took myself to be running off HEAD on master for both puppet-tripleo and T-H-T but using the suggested workaround left my T-H-T behind it's own HEAD and behind puppet-tripleo so that I hit ceilometer redis pw issues that otherwise only appear to have been hit in upgrade scenarios where these two are out of phase.

So my new approach is just to use and install the proposed fix for this bug. Seems to be working OK for me so far.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/383825

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/379547
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=c683483d0e6285129a1fdc65ce110034243f7e75
Submitter: Jenkins
Branch: master

commit c683483d0e6285129a1fdc65ce110034243f7e75
Author: Mathieu Bultel <email address hidden>
Date: Thu Sep 29 17:25:38 2016 +0200

    Download templates from swift before processing with heatclient

    Currently we hit an issue in update and upgrade, when heatclieant
    try to process the templates and failed because there was some
    missing unrendered template files, ie : post.j2.yaml

    Co-Authored-By: Steven Hardy <email address hidden>
    Change-Id: If95825e7df5d2c0e6cbe3575d06d57db1e8182da
    Closes-Bug: #1624727

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/383825
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=e5701d0673574fe11a529568e8faa78a87bf4281
Submitter: Jenkins
Branch: stable/newton

commit e5701d0673574fe11a529568e8faa78a87bf4281
Author: Mathieu Bultel <email address hidden>
Date: Thu Sep 29 17:25:38 2016 +0200

    Download templates from swift before processing with heatclient

    Currently we hit an issue in update and upgrade, when heatclieant
    try to process the templates and failed because there was some
    missing unrendered template files, ie : post.j2.yaml

    Co-Authored-By: Steven Hardy <email address hidden>
    Change-Id: If95825e7df5d2c0e6cbe3575d06d57db1e8182da
    Closes-Bug: #1624727
    (cherry picked from commit c683483d0e6285129a1fdc65ce110034243f7e75)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-tripleoclient 5.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-tripleoclient 5.5.0 release.