(Queens) is missing from insecure registries

Bug #1799784 reported by Emilien Macchi on 2018-10-24
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Emilien Macchi

Bug Description

Initially reported here: https://bugzilla.redhat.com/show_bug.cgi?id=1641704

Description of problem:

Queens is failing during containers preparation while trying to push containers on the local registry:

clean_up UploadImage: Could not push image
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 134, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 279, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 169, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 25, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/container_image.py", line 64, in take_action
  File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 96, in upload
  File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 388, in run_tasks
    result = self.upload_image(*first)
  File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 184, in upload_image
    DockerImageUploader._push(dockerc, new_repo, tag=tag)
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 214, in wrapped_f
    return self.call(f, *args, **kw)
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 295, in call
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 265, in iter
    raise RetryError(fut).reraise()
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 344, in reraise
    raise self.last_attempt.result()
  File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 422, in result
    return self.__get_result()
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 298, in call
    result = fn(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 217, in _push
    raise ImageUploaderException('Could not push image %s' % image)
ImageUploaderException: Could not push image

The command used to complete the action is:

openstack overcloud container image upload --debug --config-file /home/stack/overcloud_containers.yaml

this command was used to generate overcloud_containers.yaml file:

openstack overcloud container image prepare \
    --output-images-file /home/stack/overcloud_containers.yaml \
    --output-env-file /home/stack/containers-default-parameters.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \
    --namespace docker-registry.engineering.redhat.com/rhosp13 \
    --tag $BUILD_ID \
    --prefix "openstack" \
    --suffix "" \
    --set "ceph_namespace=docker.io/ceph" \
    --set "ceph_image=daemon" \
    --set "ceph_tag=v3.1.0-stable-3.1-luminous-centos-7-x86_64" \

The docker-registry.engineering.redhat.com is configured as INSECURE_REGISTRY inside /etc/sysconfig/docker.

Version-Release number of selected component (if applicable):

Puddle 2018-10-02.1

How reproducible:

The failure is consistent.

Steps to Reproduce:
1. Just generate the overcloud_containers.yaml file and launch the above openstack overcloud container image upload command

Actual results:

Could not push any image.

Expected results:

All the images pushed correctly.

RCA: should also be an insecure registry in /etc/sysconfig/docker.

To do this the undercloud.conf needs the following before the undercloud is installed:
docker_insecure_registries =, (...)

In Rocky, we added automatically, but instack based Queens doesn't. We can do one of the following:
- Document that they must always manually add docker_insecure_registries = to undercloud.conf
- Propose a fix to instack-undercloud to append local_ip:8787 DOCKER_INSECURE_REGISTRIES if it isn't there

Fix proposed to branch: master
Review: https://review.openstack.org/613166

Changed in tripleo:
status: Triaged → In Progress

Change abandoned by Emilien Macchi (<email address hidden>) on branch: master
Review: https://review.openstack.org/613166
Reason: https://review.openstack.org/#/c/613406/

Change abandoned by Juan Antonio Osorio Robles (<email address hidden>) on branch: stable/queens
Review: https://review.openstack.org/613406
Reason: clearing up the gate to free up resources.

Reviewed: https://review.openstack.org/613406
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=0fd8cd3f30e2c3eb81aae2a726aa661f8cf925e1
Submitter: Zuul
Branch: stable/queens

commit 0fd8cd3f30e2c3eb81aae2a726aa661f8cf925e1
Author: Emilien Macchi <email address hidden>
Date: Wed Oct 24 17:09:58 2018 -0400

    [queens-only] include local_ip & undercloud_admin_host in docker_insecure_registries

    This patch includes by default the local_ip and undercloud_admin_host in
    docker_insecure_registries parameter, that can be missed but actually
    needed in some cases.

    This code was done in tripleoclient for the containerized undercloud in
    Rocky, so we just need to port it here for stable versions, to have the
    same behavior.

    Change-Id: I03be8381ec776fd769139c79b85470acedc64d2b
    Closes-Bug: #1799784

tags: added: in-stable-queens
Changed in tripleo:
milestone: stein-1 → stein-2
Changed in tripleo:
milestone: stein-2 → stein-3
Changed in tripleo:
status: In Progress → Fix Released

This issue was fixed in the openstack/instack-undercloud 8.4.7 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers