Prepare command may output empty files

[Martin André]

Since we implemented the --environment-directory option [1] to the 'openstack overcloud container image prepare' command, there is a possibility it outputs empty files when calling the command without additional options.

Previously, calling the command without options would render the template at [2] without filtering the images. However, since we defaulted the --environment-directory to ~/.tripleo/environments to match the deploy command, this could enable the filtering in case this directory contains an environment file even though we're not explicitly asking for it.

One workaround is to always pass the roles_data file and heat environment files that are going to be used in the deployment to the 'prepare' command to get a slick file with just the necessary images, or point the --environment-directory to an empty directory to keep generating for all images.

[1] https://review.openstack.org/#/c/511521/
[2] https://github.com/openstack/tripleo-common/blob/master/container-images/overcloud_containers.yaml.j2

[Mike Bayer (zzzeek)]

the workaround to specify --environment-directory does not work, at least in queens, the .tripleo/environment directory is still being read. A file is being added here when one uses the upload-puppet-modules utility:

# normal undercloud
(undercloud) [stack@s1undercloud-0 ~]$ openstack overcloud container image prepare
container_images:
- imagename: docker.io/tripleomaster/centos-binary-aodh-api:current-tripleo
- imagename: docker.io/tripleomaster/centos-binary-aodh-evaluator:current-tripleo

# ..etc

# then upload-puppet-modules on plain puppet files:

(undercloud) [stack@s1undercloud-0 ~]$ mkdir -p tripleo-patch/puppet-tripleo
(undercloud) [stack@s1undercloud-0 ~]$ cp -R /usr/share/openstack-puppet/modules/tripleo tripleo-patch/puppet-tripleo/
(undercloud) [stack@s1undercloud-0 ~]$ upload-puppet-modules -d tripleo-patch/puppet-tripleo
Creating tarball...
Tarball created.
Creating new Swift Temp-URL-Key for container: overcloud-artifacts
Creating heat environment file: /home/stack/.tripleo/environments/puppet-modules-url.yaml
Uploading file to swift: /tmp/puppet-modules-h0MEFmT/puppet-modules.tar.gz
+-----------------------+---------------------+----------------------------------+
| object | container | etag |
+-----------------------+---------------------+----------------------------------+
| puppet-modules.tar.gz | overcloud-artifacts | 096bca64ab6da1058096d64d1840e66e |
+-----------------------+---------------------+----------------------------------+
Upload complete.

# now image prepare is non-functional:

(undercloud) [stack@s1undercloud-0 ~]$ openstack overcloud container image prepare
container_images: []

# same if you use --environment-directory:

(undercloud) [stack@s1undercloud-0 ~]$ openstack overcloud container image prepare --environment-directory=/tmp/
container_images: []

# remove the puppet-modules-url.yaml file, it works again:

(undercloud) [stack@s1undercloud-0 ~]$ mv .tripleo/environments/puppet-modules-url.yaml .
(undercloud) [stack@s1undercloud-0 ~]$ openstack overcloud container image prepare --environment-directory=./foo/
container_images:
- imagename: docker.io/tripleomaster/centos-binary-aodh-api:current-tripleo
- imagename: docker.io/tripleomaster/centos-binary-aodh-evaluator:current-tripleo
- imagename: docker.io/tripleomaster/centos-binary-aodh-listener:current-tripleo

I am at least super glad I found this issue because it's been driving me nuts for weeks.

[Steve Baker (steve-stevebaker)]

I'm going to use this bug for a specific problem I've found.

If the environment passed to prepare doesn't have a complete resource registry, many enabled services will be filtered out. This is caused by the build_service_filter function which is intended to only filter out services which are known to be not containerized.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/597726

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/599264

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/597726
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=a26410ff9d32c05d7d59e1594510bf3f154f5f64
Submitter: Zuul
Branch: master

commit a26410ff9d32c05d7d59e1594510bf3f154f5f64
Author: Steve Baker <email address hidden>
Date: Thu Aug 30 10:42:27 2018 +1200

    Handle sparse resource registry in build_service_filter

    During image prepare, the intent of build_service_filter is to filter
    out services which are *known* to be not containerized, either because
    they are puppet services, or have been mapped to 'OS::Heat::None'.

    However if a partial resource registry is passed in, then this was
    also filtering out the services with missing resource_registry
    mappings. Since we're now assuming services are containerized by
    default, this behaviour was cause many of the issues with missing or
    no image parameters being generated.

    Closes-Bug: #1727347
    Change-Id: Ie65038e3ed73a77aeec9178231ab776ed26a2d84

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/599264
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=ea12e6994f2d5dfbd7850fd54e765e9d10833709
Submitter: Zuul
Branch: stable/rocky

commit ea12e6994f2d5dfbd7850fd54e765e9d10833709
Author: Steve Baker <email address hidden>
Date: Thu Aug 30 10:42:27 2018 +1200

    Handle sparse resource registry in build_service_filter

    During image prepare, the intent of build_service_filter is to filter
    out services which are *known* to be not containerized, either because
    they are puppet services, or have been mapped to 'OS::Heat::None'.

    However if a partial resource registry is passed in, then this was
    also filtering out the services with missing resource_registry
    mappings. Since we're now assuming services are containerized by
    default, this behaviour was cause many of the issues with missing or
    no image parameters being generated.

    Closes-Bug: #1727347
    Change-Id: Ie65038e3ed73a77aeec9178231ab776ed26a2d84
    (cherry picked from a26410ff9d32c05d7d59e1594510bf3f154f5f64)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 9.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 10.0.0 release.