(Queens) 192.168.24.1:8787 is missing from insecure registries

[Emilien Macchi]

Initially reported here: https://bugzilla.redhat.com/show_bug.cgi?id=1641704

Description of problem:

Queens is failing during containers preparation while trying to push containers on the local registry:

clean_up UploadImage: Could not push image 192.168.24.1:8787/rhosp13/openstack-swift-proxy-server
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 134, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 279, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 169, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 25, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/container_image.py", line 64, in take_action
    uploader.upload()
  File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 96, in upload
    uploader.run_tasks()
  File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 388, in run_tasks
    result = self.upload_image(*first)
  File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 184, in upload_image
    DockerImageUploader._push(dockerc, new_repo, tag=tag)
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 214, in wrapped_f
    return self.call(f, *args, **kw)
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 295, in call
    start_time=start_time)
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 265, in iter
    raise RetryError(fut).reraise()
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 344, in reraise
    raise self.last_attempt.result()
  File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 422, in result
    return self.__get_result()
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 298, in call
    result = fn(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 217, in _push
    raise ImageUploaderException('Could not push image %s' % image)
ImageUploaderException: Could not push image 192.168.24.1:8787/rhosp13/openstack-swift-proxy-server

The command used to complete the action is:

openstack overcloud container image upload --debug --config-file /home/stack/overcloud_containers.yaml

this command was used to generate overcloud_containers.yaml file:

openstack overcloud container image prepare \
    --output-images-file /home/stack/overcloud_containers.yaml \
    --output-env-file /home/stack/containers-default-parameters.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \
    --namespace docker-registry.engineering.redhat.com/rhosp13 \
    --tag $BUILD_ID \
    --prefix "openstack" \
    --suffix "" \
    --set "ceph_namespace=docker.io/ceph" \
    --set "ceph_image=daemon" \
    --set "ceph_tag=v3.1.0-stable-3.1-luminous-centos-7-x86_64" \
    --push-destination 192.168.24.1:8787

The docker-registry.engineering.redhat.com is configured as INSECURE_REGISTRY inside /etc/sysconfig/docker.

Version-Release number of selected component (if applicable):

Puddle 2018-10-02.1

How reproducible:

The failure is consistent.

Steps to Reproduce:
1. Just generate the overcloud_containers.yaml file and launch the above openstack overcloud container image upload command

Actual results:

Could not push any image.

Expected results:

All the images pushed correctly.

RCA:

192.168.24.1:8787 should also be an insecure registry in /etc/sysconfig/docker.

To do this the undercloud.conf needs the following before the undercloud is installed:
docker_insecure_registries = 192.168.24.1:8787, (...)

In Rocky, we added 192.168.24.1:8787 automatically, but instack based Queens doesn't. We can do one of the following:
- Document that they must always manually add docker_insecure_registries = 192.168.24.1:8787 to undercloud.conf
- Propose a fix to instack-undercloud to append local_ip:8787 DOCKER_INSECURE_REGISTRIES if it isn't there

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/613166

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/613406

[OpenStack Infra (hudson-openstack)]

Change abandoned by Emilien Macchi (<email address hidden>) on branch: master
Review: https://review.openstack.org/613166
Reason: https://review.openstack.org/#/c/613406/

[OpenStack Infra (hudson-openstack)]

Change abandoned by Juan Antonio Osorio Robles (<email address hidden>) on branch: stable/queens
Review: https://review.openstack.org/613406
Reason: clearing up the gate to free up resources.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/613406
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=0fd8cd3f30e2c3eb81aae2a726aa661f8cf925e1
Submitter: Zuul
Branch: stable/queens

commit 0fd8cd3f30e2c3eb81aae2a726aa661f8cf925e1
Author: Emilien Macchi <email address hidden>
Date: Wed Oct 24 17:09:58 2018 -0400

    [queens-only] include local_ip & undercloud_admin_host in docker_insecure_registries

    This patch includes by default the local_ip and undercloud_admin_host in
    docker_insecure_registries parameter, that can be missed but actually
    needed in some cases.

    This code was done in tripleoclient for the containerized undercloud in
    Rocky, so we just need to port it here for stable versions, to have the
    same behavior.

    Change-Id: I03be8381ec776fd769139c79b85470acedc64d2b
    Closes-Bug: #1799784

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/instack-undercloud 8.4.7 release.