container upload leaves images on the undercloud docker

[Steve Baker]

The "openstack overcloud container image upload" copies images from one registry to another by doing a docker pull then a docker push. This leaves the images on the local (undercloud) docker storage. Undercloud nodes may not be specced to have enough storage for this extra data indefinitely.

I think the upload command needs a --cleanup argument which will delete the pulled images from the local docker service after all push operations are complete.

In the long term we should use a tool like skopeo[1] which can do image transfers without going through the local docker service, but this would need to be packaged and blessed as the standard tool first.

[1] https://github.com/projectatomic/skopeo

[Martin André (mandre)]

Great, then we can remove the workaround for bug #1694709.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/515252

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/516457

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/519200

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/519201

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/519200
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=db09dffee6c6e927272ee2355689d9feff338d43
Submitter: Zuul
Branch: master

commit db09dffee6c6e927272ee2355689d9feff338d43
Author: Steve Baker <email address hidden>
Date: Wed Nov 22 11:53:47 2017 +1300

    Memoize docker uploader

    This change avoids recreating the DockerImageUploader for every
    upload. Reusing the same DockerImageUploader will allow state to be
    tracked so that concurrency and cleanup can be implemented within the
    uploader.

    Change-Id: I4cffdea9860cf98f19bc318a8fb4d929d6463fef
    Partial-Bug: #1708965

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/519201
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=7e0564106e7b01f6516c22b95082b04bc0c8fc63
Submitter: Zuul
Branch: master

commit 7e0564106e7b01f6516c22b95082b04bc0c8fc63
Author: Steve Baker <email address hidden>
Date: Mon Nov 13 10:06:58 2017 +1300

    Implement post-upload cleanup of docker images

    The docker uploader will leave a copy of all uploaded images in the
    local docker storage. This change will track those images and delete
    them from the local docker after all uploads are complete.

    If a local image is in use (for example, an already deployed
    containerised undercloud) then the delete will fail. In this case,
    only a warning is logged.

    Change-Id: Ic0424638b9ddbf77e10cfe936d0b96ff2da1a59e
    Closes-Bug: #1708965
    Closes-Bug: #1694709

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 8.3.0 release.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Steve Baker (<email address hidden>) on branch: master
Review: https://review.openstack.org/515252

[OpenStack Infra (hudson-openstack)]

Change abandoned by Steve Baker (<email address hidden>) on branch: master
Review: https://review.openstack.org/516457

[David Hill (david-hill-ubisoft)]

FYI : This breaks the local docker registry if the overcloud nodes are not able to reach the outside world as the docker images are downloaded, uploaded to the registry and then... they're all deleted!

[Bogdan Dobrelya (bogdando)]

The "docker images" command does not actually list the images in the local registry, it just lists the images in the local docker, which is essentially cached images left over from the copy operation. AFAIK, only those cached images is the subject for the --cleanup command actions supported. The registry contents may be checked with something like:

skopeo inspect --tls-verify=false docker://<ip>:8787/<namespace>/<container-registry-name>:<container-tag>

or

curl -s <ip>:8787/v2/_catalog | python -mjson.tool