tripleo-container-tag image pulling failing in upgrade_tasks

[Jose Luis Franco]

The master to master upgrades job is failing during "overcloud upgrade run --nodes Controller" step with the following error:

2019-01-31 10:38:53 | TASK [tripleo-container-tag : Pull e4c7987d55f4 image] *************************
2019-01-31 10:38:53 | task path: /usr/share/ansible/roles/tripleo-container-tag/tasks/main.yaml:2
2019-01-31 10:38:53 | Thursday 31 January 2019 10:38:51 +0000 (0:00:00.513) 0:00:15.105 ******
2019-01-31 10:38:54 | Using module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py
2019-01-31 10:38:54 | <192.168.24.3> ESTABLISH SSH CONNECTION FOR USER: tripleo-admin
2019-01-31 10:38:54 | <192.168.24.3> SSH: EXEC ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o ControlMaster=auto -o ControlPersist=30m -o ServerAliveInterval=5 -o ServerAliveCountMax=5 -o StrictHostKeyChecking=no -o 'IdentityFile="/var/lib/mistral/74a94292-11b4-4802-aa86-0da8f621dc9e/ssh_private_key"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=tripleo-admin -o ConnectTimeout=30 -o ControlPath=/var/lib/mistral/74a94292-11b4-4802-aa86-0da8f621dc9e/ansible-ssh/82430b422b 192.168.24.3 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-dflrngcymxppnztncxjgvmjiyoqgsebk; /usr/bin/python'"'"'"'"'"'"'"'"' && sleep 0'"'"''
2019-01-31 10:38:54 | Escalation succeeded
2019-01-31 10:38:54 | <192.168.24.3> (1, '\n{"changed": true, "end": "2019-01-31 10:38:51.937846", "stdout": "Using default tag: latest\\nTrying to pull repository docker.io/library/e4c7987d55f4 ... ", "cmd": "docker pull e4c7987d55f4", "failed": true, "delta": "0:00:00.216974", "stderr": "repository docker.io/e4c7987d55f4 not found: does not exist or no pull access", "rc": 1, "invocation": {"module_args": {"warn": true, "executable": null, "_uses_shell": true, "_raw_params": "docker pull e4c7987d55f4", "removes": null, "argv": null, "creates": null, "chdir": null, "stdin": null}}, "start": "2019-01-31 10:38:51.720872", "msg": "non-zero return code"}\n', '')
2019-01-31 10:38:54 | fatal: [centos-7-rax-iad-0002293874]: FAILED! => {
2019-01-31 10:38:54 | "changed": true,
2019-01-31 10:38:54 | "cmd": "docker pull e4c7987d55f4",
2019-01-31 10:38:54 | "delta": "0:00:00.216974",
2019-01-31 10:38:54 | "end": "2019-01-31 10:38:51.937846",
2019-01-31 10:38:54 | "invocation": {
2019-01-31 10:38:54 | "module_args": {
2019-01-31 10:38:54 | "_raw_params": "docker pull e4c7987d55f4",
2019-01-31 10:38:54 | "_uses_shell": true,
2019-01-31 10:38:54 | "argv": null,
2019-01-31 10:38:54 | "chdir": null,
2019-01-31 10:38:54 | "creates": null,
2019-01-31 10:38:54 | "executable": null,
2019-01-31 10:38:54 | "removes": null,
2019-01-31 10:38:54 | "stdin": null,
2019-01-31 10:38:54 | "warn": true
2019-01-31 10:38:54 | }
2019-01-31 10:38:54 | },
2019-01-31 10:38:54 | "msg": "non-zero return code",
2019-01-31 10:38:54 | "rc": 1,
2019-01-31 10:38:54 | "start": "2019-01-31 10:38:51.720872",
2019-01-31 10:38:54 | "stderr": "repository docker.io/e4c7987d55f4 not found: does not exist or no pull access",
2019-01-31 10:38:54 | "stderr_lines": [
2019-01-31 10:38:54 | "repository docker.io/e4c7987d55f4 not found: does not exist or no pull access"
2019-01-31 10:38:54 | ],
2019-01-31 10:38:54 | "stdout": "Using default tag: latest\nTrying to pull repository docker.io/library/e4c7987d55f4 ... ",
2019-01-31 10:38:54 | "stdout_lines": [
2019-01-31 10:38:54 | "Using default tag: latest",
2019-01-31 10:38:54 | "Trying to pull repository docker.io/library/e4c7987d55f4 ... "
2019-01-31 10:38:54 | ]
2019-01-31 10:38:54 | }

Log: http://logs.openstack.org/25/607525/9/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/f527802/logs/undercloud/home/zuul/overcloud_upgrade_run_Controller.log.txt.gz

Lately, the pacemaker managed services got their container image tagging tasks moved into an execution of the role tripleo-container-tag (https://github.com/openstack/tripleo-heat-templates/commit/ef6d7cf1646c897347ad4e22ea9c959402c83d5a#diff-f964751140aa8663665f2fa0463f5a6b), this ansible role contains two tasks, pulling the image we want to tag and tag it. However, in the case of the upgrade_tasks, we can't pull the input passed into {{ container_image }} as we're passing the container_image_id, not the image name: https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/pacemaker/database/mysql.yaml#L370 , so when the role tries to pull a image id, it fails as it can't find any image by that name.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/634239

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/634243

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/634239
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=a3b7185caedbc4499a6fab298f4510b35bc5e9e2
Submitter: Zuul
Branch: master

commit a3b7185caedbc4499a6fab298f4510b35bc5e9e2
Author: Jose Luis Franco Arza <email address hidden>
Date: Thu Jan 31 15:02:21 2019 +0100

    Make the image pulling optional for tripleo-container-tag role.

    The role makes possible the re-use of code at the time of tagging the
    container images. However, the tagging varies whether we are deploying
    the services or upgrading. In upgrade tasks we use this role to tag the
    new container image (from the version we're upgrading to) into the image
    being used by pacemaker. So, the value for the container_image
    parameter is an image id, not an image name, which can't be pulled.
    This patch adds a new parameter, pull_image, which is set to true by
    default and will allow us to skip the image pulling for the upgrade_tasks
    case.

    Change-Id: I6fc69f122b3f48064ed831cd622a617a02a28e02
    Related-Bug: #1814104

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/634243
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=58b99bf5eecb6ac427744c89186f3d0afbabc917
Submitter: Zuul
Branch: master

commit 58b99bf5eecb6ac427744c89186f3d0afbabc917
Author: Jose Luis Franco Arza <email address hidden>
Date: Thu Jan 31 15:23:18 2019 +0100

    Do not pull image while tagging pcmk images in upgrade_tasks.

    During the upgrad of pacemaker managed services there is
    a temporary tagging of the existing container image to the
    newly image we are going to upgrade to. The input during this
    taggins is the container image id, in contrast to what is
    passed during deploy tasks, which is the image name with its
    tag. For that reason, we can't pull the image in these cases
    otherwise the role will fail as it can't find an image to pull
    from a container image id.

    Change-Id: I4fdd3a05465fa0318b4ec5c079d59f4dd80fa9f2
    Closes-Bug: #1814104

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 10.4.0 release.