[Pike Only] Wrongly rendered namespace in docker images: <class 'jinja2.utils.Namespace'>/centos-binary-ceilometer-notification:current-tripleo"

[Jose Luis Franco]

In the upgrades job from Ocata to Pike it has been observed an error when rendering the container images during the upgrade step. The overcloud upgrade command fails with the following tasks failing:

...
2018-10-05 09:40:24 | "2018-10-05 09:39:38,539 WARNING: 144091 -- retrying pulling image: <class 'jinja2.utils.Namespace'>/centos-binary-cron:current-tripleo",
2018-10-05 09:40:24 | "2018-10-05 09:39:41,565 WARNING: 144091 -- docker pull failed: Error parsing reference: \"<class 'jinja2.utils.Namespace'>/centos-binary-cron:current-tripleo\" is not a valid repository/tag: invalid reference format",
2018-10-05 09:40:24 | "2018-10-05 09:39:41,565 WARNING: 144091 -- retrying pulling image: <class 'jinja2.utils.Namespace'>/centos-binary-cron:current-tripleo",
2018-10-05 09:40:24 | "2018-10-05 09:39:41,565 ERROR: 144091 -- Failed to pull image: <class 'jinja2.utils.Namespace'>/centos-binary-cron:current-tripleo",
2018-10-05 09:40:24 | "2018-10-05 09:39:41,592 ERROR: 144091 -- Failed running docker-puppet.py for crond",
2018-10-05 09:40:24 | "2018-10-05 09:39:41,592 ERROR: 144091 -- /usr/bin/docker-current: Error parsing reference: \"<class 'jinja2.utils.Namespace'>/centos-binary-cron:current-tripleo\" is not a valid repository/tag: invalid reference format.",
2018-10-05 09:40:24 | "2018-10-05 09:39:41,592 INFO: 144091 -- Finished processing puppet configs",
2018-10-05 09:40:24 | "2018-10-05 09:39:51,697 INFO: 144089 -- Removing container: docker-puppet-neutron",
2018-10-05 09:40:24 | "2018-10-05 09:39:51,756 INFO: 144089 -- Finished processing puppet configs",
2018-10-05 09:40:24 | "2018-10-05 09:39:51,757 ERROR: 144088 -- ERROR configuring haproxy",
2018-10-05 09:40:24 | "2018-10-05 09:39:51,757 ERROR: 144088 -- ERROR configuring nova_placement",
2018-10-05 09:40:24 | "2018-10-05 09:39:51,757 ERROR: 144088 -- ERROR configuring clustercheck",
2018-10-05 09:40:24 | "2018-10-05 09:39:51,757 ERROR: 144088 -- ERROR configuring mysql",
2018-10-05 09:40:24 | "2018-10-05 09:39:51,757 ERROR: 144088 -- ERROR configuring rabbitmq",
2018-10-05 09:40:24 | "2018-10-05 09:39:51,757 ERROR: 144088 -- ERROR configuring iscsid",
2018-10-05 09:40:24 | "2018-10-05 09:39:51,757 ERROR: 144088 -- ERROR configuring heat",
2018-10-05 09:40:24 | "2018-10-05 09:39:51,757 ERROR: 144088 -- ERROR configuring glance_api",
2018-10-05 09:40:24 | "2018-10-05 09:39:51,758 ERROR: 144088 -- ERROR configuring keystone",
2018-10-05 09:40:24 | "2018-10-05 09:39:51,758 ERROR: 144088 -- ERROR configuring nova",
2018-10-05 09:40:24 | "2018-10-05 09:39:51,758 ERROR: 144088 -- ERROR configuring horizon",
2018-10-05 09:40:24 | "2018-10-05 09:39:51,758 ERROR: 144088 -- ERROR configuring heat_api",
2018-10-05 09:40:24 | "2018-10-05 09:39:51,758 ERROR: 144088 -- ERROR configuring heat_api_cfn",
2018-10-05 09:40:24 | "2018-10-05 09:39:51,758 ERROR: 144088 -- ERROR configuring memcached",
2018-10-05 09:40:24 | "2018-10-05 09:39:51,758 ERROR: 144088 -- ERROR configuring crond"
2018-10-05 09:40:24 | ]
2018-10-05 09:40:24 | }
2018-10-05 09:40:24 | to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/df8a31ae-f608-49a5-b8e7-4fed8b8ff612_playbook.retry
2018-10-05 09:40:24 |
2018-10-05 09:40:24 | PLAY RECAP *********************************************************************
2018-10-05 09:40:24 | localhost : ok=6 changed=2 unreachable=0 failed=1
2018-10-05 09:40:24 |
2018-10-05 09:40:24 | deploy_stderr: |

https://logs.rdoproject.org/21/604121/9/openstack-experimental/legacy-tripleo-ci-centos-7-containers-multinode-upgrades-pike-branch/7923780/logs/undercloud/home/zuul/overcloud_composable_failed_upgrade.log.txt.gz#_2018-10-05_09_40_24

It can be seen that the namespace used to pull the images is unrecognized:
2018-10-05 09:40:24 | "2018-10-05 09:39:41,592 ERROR: 144091 -- Failed running docker-puppet.py for crond",
2018-10-05 09:40:24 | "2018-10-05 09:39:41,592 ERROR: 144091 -- /usr/bin/docker-current: Error parsing reference: \"<class 'jinja2.utils.Namespace'>/centos-binary-cron:current-tripleo\" is not a valid repository/tag: invalid reference format.",

When going to the mistral logs, we see in the result from the container_image_prepare action, that the templates were rendered this way:
https://logs.rdoproject.org/21/604121/9/openstack-experimental/legacy-tripleo-ci-centos-7-containers-multinode-upgrades-pike-branch/7923780/logs/undercloud/var/log/mistral/engine.log.txt.gz#_2018-10-05_09_16_21_609

[Jose Luis Franco (jfrancoa)]

The cause is in the variable name used for the namespace in the container images jinja template: https://github.com/openstack/tripleo-common/blob/stable/pike/container-images/overcloud_containers.yaml.j2#L1
This variable name matches with a newly added class in Jinja2, the built-in namespace class: http://jinja.pocoo.org/docs/2.10/templates/#namespace , so as the action container_images_prepare does not pass a value for the parameter namespace [0][1], then Jinja2 considers that the assignation
{% set namespace=namespace or "tripleopike" %} refers to the built-in class jinja2.utils.Namespace and makes the substitution.
As a consequence, all namespaces for the images contain a reference to the built-in class instead of the default value for the namespace.
An easy solution is to change the variable from namespace to docker_namespace.

[0] - https://github.com/openstack/tripleo-common/blob/stable/pike/workbooks/plan_management.yaml#L271
[1] - https://github.com/openstack/tripleo-common/blob/stable/pike/tripleo_common/actions/container_images.py#L54

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/608647

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/608648

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/608647
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=a42f1e11e4bbe151ecfcdbd9d943118a23473750
Submitter: Zuul
Branch: stable/pike

commit a42f1e11e4bbe151ecfcdbd9d943118a23473750
Author: Jose Luis Franco Arza <email address hidden>
Date: Mon Oct 8 13:34:54 2018 +0200

    [Pike Only] Do not use namespace var name in docker img template.

    The namespace variable name is reserved for a jinja2 built-in
    class [0]. So, if no parameter namespace is being passed when the
    template is being rendered, we'll end up referencing the
    jinja built-in class and the container docker images will be
    written as <class 'jinja2.utils.Namespace'>/centos-binary-manila-api.
    This patch changes the variable name into docker_namespace.

    *It's needed to change the parameter name used in python-tripleoclient
    https://review.openstack.org/608648

    [0] - http://jinja.pocoo.org/docs/2.10/templates/#namespace
    Related-Bug: #1796664

    Change-Id: Ia212ad94463541e98d41211947a52e5ddd6a6686

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/608648
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=2f75b8b5964d58ddeca1d36a33b2bd4a9c8dd81d
Submitter: Zuul
Branch: stable/pike

commit 2f75b8b5964d58ddeca1d36a33b2bd4a9c8dd81d
Author: Jose Luis Franco Arza <email address hidden>
Date: Mon Oct 8 13:41:00 2018 +0200

    [Pike Only] Change docker img rendering variable to docker_namespace.

    The namespace name variable is used by a jinja2 built-in class,
    to avoid conflicts when no parameters are passed to render
    overcloud_containers.yaml.j2 we have changed the name from
    namespace into docker_namespace.

    Change-Id: Id5948211c5570dc284826f246ba07eb9472e3779
    Depends-On: https://review.openstack.org/608647
    Closes-Bug: #1796664

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/613013
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=16cdb712484b9acc6bfb1bd7c17bc592702aabb0
Submitter: Zuul
Branch: stable/pike

commit 16cdb712484b9acc6bfb1bd7c17bc592702aabb0
Author: Jose Luis Franco Arza <email address hidden>
Date: Wed Oct 24 15:10:09 2018 +0200

    [Pike Only] Remove comment related to namespace variable.

    Due to the close dependency between python-tripleoclient and
    tripleo-common packages, it's better to leave the
    overcloud_containers.yaml.j2 template working with both
    variables, docker_namespace and namespace in case the change
    in python-tripleoclient [0] would be missing.

    [0] - Id5948211c5570dc284826f246ba07eb9472e3779

    Change-Id: I53062d2b89dd89d78eefd2c481b5670c16ba9232
    Related-Bug: #1796664

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-tripleoclient 7.3.15 release.