All container images are uploaded/specified, even for services not deployed

[Steve Baker]

Doing a container image upload will transfer every known image, even for services which are not deployed. The prepare command has the --exclude argument to exclude images not required, but it would be better to exclude automatically based on what services are containerised.

To do this the following would be required:
- refresh this change https://review.openstack.org/#/c/448328/ to capture the image -> service mapping in tripleo-common/container-images/tripleo_kolla_template_overrides.j2
- add an argument to the prepare command which takes the heat docker services environment file (tripleo-heat-templates/environments/docker.yaml) and filters the image list by the deployed services.

[Steve Baker (steve-stevebaker)]

Here is the first change, to map images to services https://review.openstack.org/#/c/448328/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/494367

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/448328
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=a8139d648d72b999088c94a02b874acb68868c76
Submitter: Jenkins
Branch: master

commit a8139d648d72b999088c94a02b874acb68868c76
Author: Steve Baker <email address hidden>
Date: Wed Aug 16 13:57:49 2017 +1200

    overcloud_containers.yaml.j2 map images to services

    This change adds a services key to each entry in
    overcloud_containers.yaml.j2 which contains a list of
    tripleo-heat-templates docker resource types which consume this image.

    The intent is for the prepare command to use this plus the heat
    environments to produce an image list that only contains images for
    services actually being deployed. This targeted list will make image
    upload operations quicker, which is one issue affecting containers in
    CI.

    Change-Id: Ie8801b11921c46923b0f7c9aaba6bf524c464e82
    Partial-Bug: #1710992

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/494767

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/494367
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=cd1eb1e111779a0a335c7e28f521acc0eb18dae3
Submitter: Jenkins
Branch: master

commit cd1eb1e111779a0a335c7e28f521acc0eb18dae3
Author: Steve Baker <email address hidden>
Date: Thu Aug 17 15:45:58 2017 +1200

    Filter container images by deployed services

    This change adds the --service-environment-file/-e argument to the
    prepare command which will take heat environment files that specify
    which services are containerised.

    When specified, the resulting image entries will be filtered only by
    the containerised services discovered in the environment.

    -e is used as the short argument so that tools like quickstart can use
    the same variable to specify these environments for both the prepare
    command and the overcloud deploy command.

    Once this change merges, any calls to prepare in CI can be modified to
    reduce the number of images being transferred for 'upload' calls. For
    example, specifying -e environment/docker.yaml will reduce image
    transfers from 85 to 44.

    Change-Id: I35866fb24915ef80b86cc402015afb08cb155dcf
    Partial-Bug: #1710992
    Depends-On: #Ie8801b11921c46923b0f7c9aaba6bf524c464e82

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/494767
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=e7c7570e721f0a9600575b89309453f15b93f109
Submitter: Jenkins
Branch: master

commit e7c7570e721f0a9600575b89309453f15b93f109
Author: Steve Baker <email address hidden>
Date: Fri Aug 18 13:57:17 2017 +1200

    Filter image list by deployed services

    This change adds environment file arguments to
    "openstack overcloud container image prepare" calls so that only
    deployed images are in the generated files.

    This will lead to upload commands only transferring the required
    images to the undercloud registry, which should lead to a significant
    speedup of containerised CI jobs.

    Change-Id: I7166be7f07b5be7b4ded453b6e926396fbb1aa5a
    Depends-On: I2769f52de1c4a128a921fa2e29ce32ae9c2d57e9
    Closes-Bug: #1710992

[Steve Baker (steve-stevebaker)]

I7166be7f07b5be7b4ded453b6e926396fbb1aa5a was reverted due to breaking containerised scenarios CI, am investigating

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/496950

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/499388

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/499424

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/496950
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=2e637c58259925a5e70895f6d8b27f8c6c7c253a
Submitter: Jenkins
Branch: master

commit 2e637c58259925a5e70895f6d8b27f8c6c7c253a
Author: Steve Baker <email address hidden>
Date: Fri Aug 18 13:57:17 2017 +1200

    Filter image list by deployed services

    This change adds environment file arguments to
    "openstack overcloud container image prepare" calls so that only
    deployed images are in the generated files.

    This will lead to upload commands only transferring the required
    images to the undercloud registry, which should lead to a significant
    speedup of containerized CI jobs.

    Change-Id: I48b6b2e2be847314c45ae1d399fa70caf54ac2f4
    Closes-Bug: #1710992

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/499548

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/499424
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=0bd00bba4a99493aef01d272f380d1b359bd5d9e
Submitter: Jenkins
Branch: stable/pike

commit 0bd00bba4a99493aef01d272f380d1b359bd5d9e
Author: Steve Baker <email address hidden>
Date: Fri Jul 14 12:03:53 2017 +1200

    Filter container images by deployed services

    This change adds the --service-environment-file/-e argument to the
    prepare command which will take heat environment files that specify
    which services are containerised.

    When specified, the resulting image entries will be filtered only by
    the containerised services discovered in the environment.

    -e is used as the short argument so that tools like quickstart can use
    the same variable to specify these environments for both the prepare
    command and the overcloud deploy command.

    Once this change merges, any calls to prepare in CI can be modified to
    reduce the number of images being transferred for 'upload' calls. For
    example, specifying -e environment/docker.yaml will reduce image
    transfers from 85 to 44.

    (cherry picked from commit 9ff9b7c663b9e63b17b62f3aa9fe48a4dfd5b407)
                    and commit cd1eb1e111779a0a335c7e28f521acc0eb18dae3)
    Change-Id: I35866fb24915ef80b86cc402015afb08cb155dcf
    Partial-Bug: #1710992
    Partial-Bug: #1714270
    Depends-On: #Ie8801b11921c46923b0f7c9aaba6bf524c464e82

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/499548
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=ac8b61df5b5df1f61c416369a6362d0ee0505803
Submitter: Jenkins
Branch: master

commit ac8b61df5b5df1f61c416369a6362d0ee0505803
Author: Bogdan Dobrelya <email address hidden>
Date: Thu Aug 31 12:20:31 2017 +0200

    Fix filter image list by deployed services

    Always specify t-h-t templates for the undercloud
    deploy container images prepare command.

    Follow-up I48b6b2e2be847314c45ae1d399fa70caf54ac2f4
    Related-Bug: #1710992

    Change-Id: I6abf8d3d238f19b02effc60b727db80293516734
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/500126

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/499388
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=6adb3f1f6b626dca13540915813d080535146793
Submitter: Jenkins
Branch: master

commit 6adb3f1f6b626dca13540915813d080535146793
Author: Steve Baker <email address hidden>
Date: Thu Aug 31 11:21:28 2017 +1200

    Add missing OVN container service entries

    These will be required when deploying OVN and filtering image
    operations by deployed services.

    Change-Id: I4ae8ba9f2c5d8101b7c2f2db64d636ae8e49bc8e
    Partial-Bug: #1710992

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/500582

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/500582
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=70d16524dad64e7b22615e80acb77f6eed0d94a6
Submitter: Jenkins
Branch: stable/pike

commit 70d16524dad64e7b22615e80acb77f6eed0d94a6
Author: Steve Baker <email address hidden>
Date: Thu Aug 31 11:21:28 2017 +1200

    Add missing OVN container service entries

    These will be required when deploying OVN and filtering image
    operations by deployed services.

    Change-Id: I4ae8ba9f2c5d8101b7c2f2db64d636ae8e49bc8e
    Partial-Bug: #1710992
    (cherry picked from commit 6adb3f1f6b626dca13540915813d080535146793)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/500126
Committed: https://git.openstack.org/cgit/openstack/tripleo-docs/commit/?id=e3a14b282157c74faae0323d8e9cbe9924a4aec7
Submitter: Jenkins
Branch: master

commit e3a14b282157c74faae0323d8e9cbe9924a4aec7
Author: Martin André <email address hidden>
Date: Fri Sep 1 18:32:01 2017 +0200

    Document how to upload only the needed container images

    It is possible to limit the container images that are pulled from the
    registry by specifying the environments and role files to the prepare
    command.

    Change-Id: Id815e865d68e47d05a408b987c638ec917ca4dfe
    Related-Bug: #1710992

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-quickstart-extras 2.1.1 release.