Openstack docker images are not updated on overcloud nodes during Rocky minor upgrade

[Anton Antonov]

Description:

  After running a minor update on Rocky RDO I noticed that on overcloud nodes docker images were not updated. I was able to update the images only if I stop docker containers and delete images manually. Then the new docker images versions were successfully pulled from docker.io/tripleomaster

Steps to reproduce:

1. Update undercloud
```
yum install -y https://trunk.rdoproject.org/centos7/current/python2-tripleo-repos-0.0.1-0.20190612102512.9b3187e.el7.noarch.rpm

tripleo-repos -b rocky current ceph
yum -y update
openstack undercloud upgrade
```

2. Prepare images and overcloud update. The default docker.io/tripleomaster registry is used for docker images:

```
openstack tripleo container image prepare default \
  --output-env-file /home/stack/vxrdo/templates/containers-prepare-parameter.yaml

openstack overcloud update prepare ... \
  -e /home/stack/vxrdo/templates/containers-prepare-parameter.yaml

openstack overcloud external-update run --tags container_image_prepare

```

3. Update overcloud node(s)
```
openstack overcloud update run --nodes ...
```

Expected results:

  All packages and docker images are updated on overcloud node(s).

Actual results:

- On ap14-ctrl1 we have old images used after update:
```
[root@ap14-ctrl1 ~]# docker images |grep neutron
docker.io/tripleorocky/centos-binary-neutron-l3-agent current-tripleo 62beb23dba1d 8 weeks ago 1.35 GB
docker.io/tripleorocky/centos-binary-neutron-server current-tripleo 2eb3e299a9f5 8 weeks ago 1.21 GB
docker.io/tripleorocky/centos-binary-neutron-dhcp-agent current-tripleo f3efac5d182f 8 weeks ago 1.34 GB
docker.io/tripleorocky/centos-binary-neutron-openvswitch-agent current-tripleo 29c2d284e6a2 8 weeks ago 1.15 GB
docker.io/tripleorocky/centos-binary-neutron-metadata-agent current-tripleo 2efa7fe56168 8 weeks ago 1.11 GB

[root@ap14-ctrl1 ~]# docker inspect neutron_ovs_agent |grep -i image
        "Image": "sha256:29c2d284e6a2dbb9d82c6d03a89e4dbe39bf8beb9cb84232bbd6ca5e05aaf8a1",
            "Image": "docker.io/tripleorocky/centos-binary-neutron-openvswitch-agent:current-tripleo",
```

- On ap12-ctrl3 where old images were deleted before the update:
```
[root@ap12-ctrl3 ~]# docker images |grep neutron
docker.io/tripleorocky/centos-binary-neutron-server current-tripleo 29871c04dda1 31 hours ago 1.26 GB
docker.io/tripleorocky/centos-binary-neutron-dhcp-agent current-tripleo 29b6c062080a 31 hours ago 1.4 GB
docker.io/tripleorocky/centos-binary-neutron-l3-agent current-tripleo 717de0fb89e3 31 hours ago 1.41 GB
docker.io/tripleorocky/centos-binary-neutron-metadata-agent current-tripleo 6d8cacc96d9a 31 hours ago 1.17 GB
docker.io/tripleorocky/centos-binary-neutron-openvswitch-agent current-tripleo dc2ed8e405a3 31 hours ago 1.2 GB

[root@ap12-ctrl3 ~]# docker inspect neutron_ovs_agent |grep -i image
        "Image": "sha256:dc2ed8e405a3a03d23ae2216a92e14538503b184c96530aac7fc4c6183c6c505",
            "Image": "docker.io/tripleorocky/centos-binary-neutron-openvswitch-agent:current-tripleo",
```

  There are no connectivity issues from ap14-ctrl1 to docker.io/tripleomaster. Please have a look at the full output attached. Only docker images manually deleted before the update were pulled from the docker images registry.

Environment:
```
(undercloud) [stack@undercloud ~]$ rpm -qa|grep tripleo
ansible-role-tripleo-modify-image-1.0.1-0.20190612031350.367f3e0.el7.noarch
puppet-tripleo-9.4.1-0.20190608052722.b56985d.el7.noarch
openstack-tripleo-validations-9.3.2-0.20190611232838.8a84c01.el7.noarch
openstack-tripleo-image-elements-9.1.1-0.20190604163040.a66a60d.el7.noarch
openstack-tripleo-heat-templates-9.3.1-0.20190613210447.11e842d.el7.noarch
python-tripleoclient-heat-installer-10.6.2-0.20190607003405.8d9fa8a.el7.noarch
ansible-tripleo-ipsec-9.1.1-0.20190513182453.ffe104c.el7.noarch
python2-tripleo-common-9.5.1-0.20190611182931.83db258.el7.noarch
openstack-tripleo-puppet-elements-9.0.2-0.20190605172111.81729a1.el7.noarch
python2-tripleo-repos-0.0.1-0.20190612102607.9b3187e.el7.noarch
openstack-tripleo-common-containers-9.5.1-0.20190611182931.83db258.el7.noarch
openstack-tripleo-common-9.5.1-0.20190611182931.83db258.el7.noarch
python-tripleoclient-10.6.2-0.20190607003405.8d9fa8a.el7.noarch
```

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/676102

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/676415

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/676427

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jose Luis Franco (<email address hidden>) on branch: master
Review: https://review.opendev.org/676102
Reason: This issue should get fixed by https://review.opendev.org/#/q/topic:fix_container_image_prepare+(status:open+OR+status:merged)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/676899

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/676904

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/677776

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/677777

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/676415
Committed: https://git.openstack.org/cgit/openstack/tripleo-validations/commit/?id=db30779959c78208b39e716c946152ae8f7e904c
Submitter: Zuul
Branch: master

commit db30779959c78208b39e716c946152ae8f7e904c
Author: Jose Luis Franco Arza <email address hidden>
Date: Wed Aug 14 16:16:15 2019 +0200

    Add undercloud-key-file and ssh_network options.

    To enable the execution via ssh into the Undercloud node it's
    required to pass also the ansible_ssh_user private key. This
    options is part of the tripleo_common inventory.py, but defaults
    to None.
    This patch allows to pass this parameter in case it's needed.

    Also, this patch allows to pass a different value for the host_network
    parameter in TripleoInventory instantiation. Setting it by default to
    ctlplane.

    Change-Id: Id303f356c1b06906aa5aae43b0032bb80a712508
    Related-Bug: #1832932

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/679054

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/679061

[OpenStack Infra (hudson-openstack)]

Change abandoned by Athlan-Guyot sofer (<email address hidden>) on branch: stable/stein
Review: https://review.opendev.org/677776
Reason: not needed anymore for testing.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/676427
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=28a3d020618f633608de2657a8589c32fe44a964
Submitter: Zuul
Branch: master

commit 28a3d020618f633608de2657a8589c32fe44a964
Author: Jose Luis Franco Arza <email address hidden>
Date: Wed Aug 14 16:32:09 2019 +0200

    Pass undercloud's private key to inventory used in package_update.

    As the Ansible playbook is being executed from the mistral_executor
    container, we need to establish a conection to the Undercloud via ssh
    not local as it was being done up to now. In order to be able to ssh
    into the host we need to provide with the tripleo-ansible's user ssh
    key.

    Depends-On: https://review.opendev.org/#/c/676415/

    Closes-Bug: #1832932
    Change-Id: I20948d49b3023dc76acff8779bbe182886e55838

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/677777
Committed: https://git.openstack.org/cgit/openstack/tripleo-validations/commit/?id=83f84fa6cf2536b729b589e9d57957c4fbad6b63
Submitter: Zuul
Branch: stable/stein

commit 83f84fa6cf2536b729b589e9d57957c4fbad6b63
Author: Jose Luis Franco Arza <email address hidden>
Date: Wed Aug 14 16:16:15 2019 +0200

    Add undercloud-key-file and ssh_network options.

    To enable the execution via ssh into the Undercloud node it's
    required to pass also the ansible_ssh_user private key. This
    options is part of the tripleo_common inventory.py, but defaults
    to None.
    This patch allows to pass this parameter in case it's needed.

    Also, this patch allows to pass a different value for the host_network
    parameter in TripleoInventory instantiation. Setting it by default to
    ctlplane.

    Change-Id: Id303f356c1b06906aa5aae43b0032bb80a712508
    Related-Bug: #1832932
    (cherry picked from commit db30779959c78208b39e716c946152ae8f7e904c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/679061
Committed: https://git.openstack.org/cgit/openstack/tripleo-validations/commit/?id=b60bc846444cff42abcc8afd913e4cf34fc33524
Submitter: Zuul
Branch: master

commit b60bc846444cff42abcc8afd913e4cf34fc33524
Author: Jose Luis Franco Arza <email address hidden>
Date: Wed Aug 28 15:52:30 2019 +0200

    Pass undercloud_connection in TripleoInventory instantiation.

    The parameter undercloud_connection is being available as
    an input option in the script, however it was not being passed
    into the TripleoInventory object instantiation.

    Change-Id: Ife819b1d807946c8e53deaacbb8e4f9361009f6e
    Closes-Bug: #1832932

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/679054
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=9b9b5aaadc400eb6f998ccc22506a50bdd6c84ab
Submitter: Zuul
Branch: stable/stein

commit 9b9b5aaadc400eb6f998ccc22506a50bdd6c84ab
Author: Jose Luis Franco Arza <email address hidden>
Date: Wed Aug 14 16:32:09 2019 +0200

    Pass undercloud's private key to inventory used in package_update.

    As the Ansible playbook is being executed from the mistral_executor
    container, we need to establish a conection to the Undercloud via ssh
    not local as it was being done up to now. In order to be able to ssh
    into the host we need to provide with the tripleo-ansible's user ssh
    key.

    Depends-On: https://review.opendev.org/#/c/677777/

    Closes-Bug: #1832932
    Change-Id: I20948d49b3023dc76acff8779bbe182886e55838

    (cherry picked from commit 28a3d020618f633608de2657a8589c32fe44a964)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/679243

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/679243
Committed: https://git.openstack.org/cgit/openstack/tripleo-validations/commit/?id=e62b8ee8785dd0e58b5caaf62e885ec62553fe1a
Submitter: Zuul
Branch: stable/stein

commit e62b8ee8785dd0e58b5caaf62e885ec62553fe1a
Author: Jose Luis Franco Arza <email address hidden>
Date: Wed Aug 28 15:52:30 2019 +0200

    Pass undercloud_connection in TripleoInventory instantiation.

    The parameter undercloud_connection is being available as
    an input option in the script, however it was not being passed
    into the TripleoInventory object instantiation.

    Change-Id: Ife819b1d807946c8e53deaacbb8e4f9361009f6e
    Closes-Bug: #1832932
    (cherry picked from commit b60bc846444cff42abcc8afd913e4cf34fc33524)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-tripleoclient 11.5.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-validations 10.5.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-tripleoclient 12.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-validations 11.2.0 release.

[OpenStack Infra (hudson-openstack)]

Change abandoned by wes hayutin (<email address hidden>) on branch: master
Review: https://review.opendev.org/676904
Reason: https://specs.openstack.org/openstack/tripleo-specs/specs/policy/patch-abandonment.html

[OpenStack Infra (hudson-openstack)]

Change abandoned by wes hayutin (<email address hidden>) on branch: master
Review: https://review.opendev.org/676899
Reason: https://specs.openstack.org/openstack/tripleo-specs/specs/policy/patch-abandonment.html