Could not fetch contents for file:///home/stack/tripleo-heat-templates/puppet/post.yaml

Bug #1624727 reported by Michele Baldessari on 2016-09-17
30
This bug affects 5 people
Affects Status Importance Assigned to Milestone
tripleo
Critical
Steven Hardy

Bug Description

While testing upgrades from M->N, right after I upgraded my undercloud without issues, I get this
when trying to do the first step of the overcloud ugprade (the one that sets the repos).
+ openstack overcloud deploy --templates /home/stack/tripleo-heat-templates --libvirt-type qemu --control-flavor oooq_control --compute-flavor oooq_compute --ceph-storage-flavor oooq_ceph --timeout 75 --ntp-server clock.redhat.com --control-scale 3 --neutron-network-type vxlan --neutron-tunnel-types vxlan -e /home/stack/tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /home/stack/tripleo-heat-templates/environments/network-isolation.yaml -e /home/stack/tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e /home/stack/network-environment.yaml -e /home/stack/tripleo-heat-templates/environments/puppet-pacemaker.yaml -e /home/stack/tripleo-heat-templates/environments/major-upgrade-pacemaker-init.yaml -e /home/stack/overcloud-repo.yaml --ntp-server clock.redhat.com -e /tmp/deploy_env.yaml
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
WARNING: openstackclient.common.exceptions is deprecated and will be removed after Jun 2017. Please use osc_lib.exceptions
Removing the current plan files
Uploading new plan files
Plan updated
Deploying templates in the directory /home/stack/tripleo-heat-templates
Could not fetch contents for file:///home/stack/tripleo-heat-templates/puppet/post.yaml

Both Tom Barron and I are hitting this. I have tried to apply the following reviews before that step:
https://review.openstack.org/#/c/371347/
https://review.openstack.org/#/c/371027
https://review.openstack.org/#/c/368760 (this one is merged but is not yet in the newton rdo repos in current-ci-repo so I had to apply it manually)

The file puppet/post.yaml does not exist. Enly post.j2.yaml

Changed in tripleo:
milestone: none → newton-rc2
Tom Barron (tpb) wrote :

Thanks for trying to apply the above reviews, Michele. I hit the issue as follows:

http://paste.fedoraproject.org/429357/14741233/

I can hit this (1) on the first deploy in a brand new environment, (2) redeploy - where I see messages about 'Removing current plan files' and 'Uploading new plan files', and (3) redeploy after first removing the overcloud plan with 'openstack overcloud plan delete overcloud'

This is on master of THT and puppet/tripleo on fresh instack and baremetal vms deployed via tripleo.sh as of about 12 hours ago.

Dougal Matthews (d0ugal) wrote :

I believe this is essentially a duplicate of https://bugs.launchpad.net/tripleo/+bug/1622683

Michele Baldessari (michele) wrote :

Thanks for looking at this Dougal. Do we need other reviews than these to get things working:
https://review.openstack.org/#/c/371347/
https://review.openstack.org/#/c/371027
https://review.openstack.org/#/c/368760

I ask because I tried and made sure I had those applied but had no success

Steven Hardy (shardy) wrote :

@michele - did you re-run openstack undercloud install after applying those patches?

That is required to reload the mistral actions/workflows from the patches touching tripleo-common, or you can do it manually like:

$ cat mistral_update.sh
#!/bin/bash
sudo mistral-db-manage populate
sleep 1
sudo systemctl restart openstack-mistral-engine
sudo systemctl restart openstack-mistral-executor
sudo systemctl restart openstack-mistral-api
sleep 1

for workbook in $(mistral workbook-list | grep tripleo | cut -f 2 -d ' '); do
    mistral workbook-delete $workbook
done
for workflow in $(mistral workflow-list | grep tripleo | cut -f 2 -d ' '); do
    mistral workflow-delete $workflow
done
for workbook in $(ls /usr/share/openstack-tripleo-common/workbooks/*); do
    mistral workbook-create $workbook
done

Changed in tripleo:
status: New → Triaged
Michele Baldessari (michele) wrote :

Thanks Steve. No I had only done the db-manage + service restart without the deletion.

If I do the suggested steps I get:
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
Removing the current plan files
Uploading new plan files
Exception updating plan: The environment is not a valid YAML mapping data type.

Trying to debug it now.

Tom Barron (tpb) wrote :

I blew away instack and baremetal vms today, yum updated, and built everything fresh. Deployed overcloud as follows:
[stack@instack ~]$ cat > ~/netapp.yaml << _EOF_
> parameter_defaults:
> ManilaNetappLogin: 'snapmirror'
> ManilaNetappPassword: 'ssssshhhhhh'
> ManilaNetappServerHostname: '111.111.111.111'
> ManilaNetappVserver: 'Vserver1'
> ManilaNetappRootVolumeAggr: 'Aggr0'
> ManilaNetappTraceFlags: 'false'
> _EOF_
[stack@instack ~]$ export THT_ROOT=/usr/share/openstack-tripleo-heat-templates
[stack@instack ~]$ openstack overcloud deploy --templates $THT_ROOT \
> -e $THT_ROOT/overcloud-resource-registry-puppet.yaml \
> -e $THT_ROOT/environments/puppet-pacemaker.yaml \
> --control-scale 3 --compute-scale 1\
> --libvirt-type qemu --ntp-server 0.fedora.pool.ntp.org\
> -e $THT_ROOT/environments/manila-netapp-config.yaml \
> -e ./netapp.yaml
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
Creating Swift container to store the plan
Creating plan from template files in: /usr/share/openstack-tripleo-heat-templates
Plan created
Deploying templates in the directory /usr/share/openstack-tripleo-heat-templates
Could not fetch contents for file:///usr/share/openstack-tripleo-heat-templates/puppet/post.yaml

This is not a re-deploy and I'm not using relative paths.

Tom Barron (tpb) wrote :

Rebuilt everything today and updated all packages but no joy.

http://paste.fedoraproject.org/432017/46694514/

Ryan Hallisey (rthall14) wrote :

The jinja2 rendering is working properly now. The issue is that heat is not finding the jinja2 files that get rendered in swift in the working THT directory. I don't know what the intended workflow is here: whether to render the templates and have heat pull the rendered files from swift or mistral downloading them to the working THT dir.

Workaround:

move into your active THT dir and run: swift download overcloud

Tom Barron (tpb) wrote :

> Workaround: move into your active THT dir and run: swift download overcloud

That workaround, in combination with 'openstack overcloud plan delete overcloud', is getting me past this blocker. Thanks!

Steven Hardy (shardy) wrote :

Ok, thanks for the feedback, definitely sounds like we have some issues due to the current workflow ordering which is:

1. Create plan
2. Plan renders j2 templates (stored in swift)
3. Tripleoclient attempts to locally resolve the files (the pre-rendered tree not the rendered one in swift).

The workaround above seems reasonable, for now, but we need this to be sorted out in tripleoclient really, because it's always going to be possible to reference environment files etc outside of the --templates directory.

Tom Barron (tpb) wrote :

Oh, and for the record, I'm using:
python-tripleoclient.noarch 5.1.1-0.20160920215837.fdbb7be.el7.centos

Michele Baldessari (michele) wrote :

I also would like to confirm that the workaround of doing "swift download overcloud" in the tht folder fixes this specific issue for me.

Fix proposed to branch: master
Review: https://review.openstack.org/379547

Changed in tripleo:
assignee: nobody → mbu (mat-bultel)
status: Triaged → In Progress
Changed in tripleo:
milestone: newton-rc2 → ocata-1
tags: added: newton-backport-potential
Changed in tripleo:
milestone: ocata-1 → newton-rc3
Tom Barron (tpb) wrote :

I did a fresh tripleo.sh install today, new packages and new instack VM, and now see a similar but different failure at overcloud deploy:

 "Error: The following files were not found: /home/stack/tripleo-heat-templates/overcloud-resource-registry-puppet.yaml"

More details at: http://paste.fedoraproject.org/439532/47532303/

The same 'swift download overcloud' into THT workaround appears to work.

mathieu bultel (mat-bultel) wrote :

Yes, the latest update of the review ^ should handle this issue also.
I put the review as WIP since there is a bunch of things to do before trying to merged it, but you can test it if you want.
I tested it on an upgrade (Mitaka to newton), looks goods for now.
I kicked the gate... lets see how it goes.

Carlos Camacho (ccamacho) wrote :

Just checking this from last week.

If using env files and if the plan is not created you will find issues. If a prev plan is created it will be "re created"

https://paste.fedoraproject.org/439078/14752428/

Tom Barron (tpb) wrote :

Just noting that although the "swift download overcloud" into T-H-T workaround "works", it modifies other files than just what is needed. In particular, I took myself to be running off HEAD on master for both puppet-tripleo and T-H-T but using the suggested workaround left my T-H-T behind it's own HEAD and behind puppet-tripleo so that I hit ceilometer redis pw issues that otherwise only appear to have been hit in upgrade scenarios where these two are out of phase.

So my new approach is just to use and install the proposed fix for this bug. Seems to be working OK for me so far.

Changed in tripleo:
assignee: mbu (mat-bultel) → Steven Hardy (shardy)

Reviewed: https://review.openstack.org/379547
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=c683483d0e6285129a1fdc65ce110034243f7e75
Submitter: Jenkins
Branch: master

commit c683483d0e6285129a1fdc65ce110034243f7e75
Author: Mathieu Bultel <email address hidden>
Date: Thu Sep 29 17:25:38 2016 +0200

    Download templates from swift before processing with heatclient

    Currently we hit an issue in update and upgrade, when heatclieant
    try to process the templates and failed because there was some
    missing unrendered template files, ie : post.j2.yaml

    Co-Authored-By: Steven Hardy <email address hidden>
    Change-Id: If95825e7df5d2c0e6cbe3575d06d57db1e8182da
    Closes-Bug: #1624727

Changed in tripleo:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/383825
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=e5701d0673574fe11a529568e8faa78a87bf4281
Submitter: Jenkins
Branch: stable/newton

commit e5701d0673574fe11a529568e8faa78a87bf4281
Author: Mathieu Bultel <email address hidden>
Date: Thu Sep 29 17:25:38 2016 +0200

    Download templates from swift before processing with heatclient

    Currently we hit an issue in update and upgrade, when heatclieant
    try to process the templates and failed because there was some
    missing unrendered template files, ie : post.j2.yaml

    Co-Authored-By: Steven Hardy <email address hidden>
    Change-Id: If95825e7df5d2c0e6cbe3575d06d57db1e8182da
    Closes-Bug: #1624727
    (cherry picked from commit c683483d0e6285129a1fdc65ce110034243f7e75)

tags: added: in-stable-newton

This issue was fixed in the openstack/python-tripleoclient 5.3.0 release.

This issue was fixed in the openstack/python-tripleoclient 5.5.0 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers