Tripleo deployed OpenShift deployment fails when using default containers-prepare-parameter.yaml

[Martin André]

Originally reported at https://bugzilla.redhat.com/show_bug.cgi?id=1654257

Steps to Reproduce:
1. Generate a default containers-prepare-parameter.yaml file with `openstack tripleo container image prepare default --local-push-destination --output-env-file containers-prepare-parameter.yaml`
2. Deploy with CNS enabled, using the generated prepare file:
   $ openstack overcloud deploy --stack openshift --templates \
        -r /home/stack/openshift_roles_data.yaml \
        -n /home/stack/network_data_openshift.yaml \
        -e /usr/share/openstack-tripleo-heat-templates/environments/network-environment.yaml \
        -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
        -e /usr/share/openstack-tripleo-heat-templates/environments/openshift.yaml \
        -e /usr/share/openstack-tripleo-heat-templates/environments/openshift-cns.yaml \
        -e /home/stack/openshift_env.yaml \
        -e /home/stack/containers-prepare-parameter.yaml

3. Wait for deployment to finish

Actual results:

openshift-ansible fails with:

TASK [openshift_storage_glusterfs : Wait for GlusterFS pods] *******************
FAILED - RETRYING: Wait for GlusterFS pods (30 retries left).
FAILED - RETRYING: Wait for GlusterFS pods (29 retries left).
FAILED - RETRYING: Wait for GlusterFS pods (28 retries left).
FAILED - RETRYING: Wait for GlusterFS pods (27 retries left).
FAILED - RETRYING: Wait for GlusterFS pods (26 retries left).
[snip]
{"waiting": {"message": "Back-off pulling image \"172.16.0.1:8787/rhgs3/rhgs-server-rhel7\"", "reason": "ImagePullBackOff"}}
[snip]

INSTALLER STATUS ***************************************************************
Initialization : Complete (0:02:09)
Health Check : Complete (0:00:21)
Node Bootstrap Preparation : Complete (0:05:24)
etcd Install : Complete (0:01:46)
Master Install : Complete (0:08:45)
Master Additional Install : Complete (0:02:05)
Node Join : Complete (0:00:51)
GlusterFS Install : In Progress (0:06:08)

Nodes try to download an image with 'latest' tag:

Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Normal Pulling 1h (x168 over 14h) kubelet, openshift-worker-1 pulling image "172.16.0.1:8787/rhgs3/rhgs-server-rhel7"
  Normal BackOff 7m (x3834 over 14h) kubelet, openshift-worker-1 Back-off pulling image "172.16.0.1:8787/rhgs3/rhgs-server-rhel7"
  Warning Failed 2m (x3855 over 14h) kubelet, openshift-worker-1 Error: ImagePullBackOff

Expected results:
Deployment succeeds without errors.

Additional info:

This is caused by the "tag_from_label: '{version}-{release}'" line in the generated containers-prepare-parameter.yaml file. The gluster images are pulled with the 'latest' tag and uploaded to the local registry with a versioned tag, eg '3.11.0-6'.

Openshift-ansible commit https://github.com/openshift/openshift-ansible/commit/0be4b2565beb92c064917627863401af7dfb73d3 deprecated the use of openshift_storage_glusterfs_*_version variables to specify the image tags. They now need to be passed directly into the openshift_storage_glusterfs_*_image variables.

As a workaround we can remove the "tag_from_label" line from containers-prepare-parameter.yaml.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/620557

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/620557
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=7c070ab11ec45a0e3572bdec060fe31fb42cb164
Submitter: Zuul
Branch: master

commit 7c070ab11ec45a0e3572bdec060fe31fb42cb164
Author: Martin André <email address hidden>
Date: Wed Nov 28 11:45:49 2018 +0100

    Fix address for glusterfs container images

    The `openshift_storage_glusterfs*_version` openshift-ansible variables
    were removed in commit [1] as a way to specify the image tags to use.
    Instead, the image URL needs to be fully specified in the
    `openshift_storage_glusterfs*_image` variables to include the tags.

    [1] https://github.com/openshift/openshift-ansible/commit/0be4b2565beb92c064917627863401af7dfb73d3

    Change-Id: Ia68ff32008c154a644af88ef5d4da21b2b368526
    Closes-Bug: #1805611

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/631705

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/631705
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=b49ce79c1be6c0152c658ae6f9229a10f3099689
Submitter: Zuul
Branch: stable/rocky

commit b49ce79c1be6c0152c658ae6f9229a10f3099689
Author: Martin André <email address hidden>
Date: Wed Nov 28 11:45:49 2018 +0100

    Fix address for glusterfs container images

    The `openshift_storage_glusterfs*_version` openshift-ansible variables
    were removed in commit [1] as a way to specify the image tags to use.
    Instead, the image URL needs to be fully specified in the
    `openshift_storage_glusterfs*_image` variables to include the tags.

    [1] https://github.com/openshift/openshift-ansible/commit/0be4b2565beb92c064917627863401af7dfb73d3

    Change-Id: Ia68ff32008c154a644af88ef5d4da21b2b368526
    Closes-Bug: #1805611
    (cherry picked from commit 7c070ab11ec45a0e3572bdec060fe31fb42cb164)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 10.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 9.3.0 release.