Docker not configured to pull from undercloud registry by default

[Steven Hardy]

I enabled logging requests from the undercloud registry like this:

[root@undercloud /]# cat /etc/docker-distribution/registry/config.yml
version: 0.1
log:
  accesslog:
    disabled: false
  fields:
    service: registry
storage:
    cache:
        layerinfo: inmemory
    filesystem:
        rootdirectory: /var/lib/registry
http:
    addr: 192.168.24.1:8787

Then I did docker pull from the undercloud, and some deployed overcloud nodes - they hit dockerhub without trying the local registry AFAICT. I assume we're passing the registry location explicitly in the heat hook (are we?! I've yet to re deploy with logging enabled), but it'd be good if we made manual operations also use the local registry where possible.

The option to do this seems to be:

--add-registry=192.168.24.1:8787 which can be appended to OPTIONS in /etc/sysconfig/docker (puppet-tripleo will need updating to do this I think).

I also wonder if, at least optionally, we should block dockerhub so we can be sure all the images are actually pulled to the local registry e.g in CI, otherwise we may be taking a performance hit without realizing? There's a --block-registry option that does this I think.

[Steven Hardy (shardy)]

Actually in my environment during an overcloud there's no access to the undercloud registry recorded at all AFAICS.

[Steve Baker (steve-stevebaker)]

Have you set the DockerNamespace param in the environment?

parameter_defaults:
  DockerNamespace: 192.168.24.1:8787/tripleoupstream # or tripleo, whatever
  DockerNamespaceIsRegistry: true

[Dan Prince (dan-prince)]

Exactly like stevebaker said. I use a local registry (not on my undercloud though). To activate it you just need to add heat parameters to your overcloud environment like:

  DockerNamespaceIsRegistry: true
  DockerNamespace: 172.19.0.2:8787/tripleoupstream

[Steven Hardy (shardy)]

I'm using /home/stack/containers-default-parameters.yaml which gets created by quickstart, it looks like:

$ cat /home/stack/containers-default-parameters.yaml
parameter_defaults:
  # NovaImage: atomic-image # FYI the team is using overcloud-full currently https://review.openstack.org/#/c/395880/
  # Defaults to 'tripleoupstream'. Specify a local docker registry
  # Example: 192.0.2.1:8787/tripleoupstream
  DockerNamespace: 192.168.24.1:8787/tripleoupstream
  # Enable local Docker registry
  DockerNamespaceIsRegistry: true

So it does appear to be configured correctly, and the service is listening on the right IP:
$ netstat -taupen | grep 8787
(No info could be read for "-p": geteuid()=1000 but you should be root.)
tcp 0 0 192.168.24.1:8787 0.0.0.0:* LISTEN 0 1352248 -

Also the environment file doesn't solve the case where an operator interacts with docker directly on the CLI (I know we probably don't want to encourage that, but it's useful for developers at least).

[Steven Hardy (shardy)]

Ok so the root cause of this (thanks to mandre for helping work through this) is we set DockerNamespaceIsRegistry to false in this environment (which is the default in the templates anyway), which if (like me) you specify the environments in an unfortunate order, the quickstart generated environment setting gets overwritten by the docker.yaml.

https://github.com/openstack/tripleo-heat-templates/blob/master/environments/docker.yaml#L57

Since this is just duplicating the template defaults, we should probably remove them from docker.yaml I think.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/465910

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/465910
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=47dfa57f5cc8a96cf0682eec226d68908819cc75
Submitter: Jenkins
Branch: master

commit 47dfa57f5cc8a96cf0682eec226d68908819cc75
Author: Steven Hardy <email address hidden>
Date: Thu May 18 10:53:10 2017 +0100

    Comment parameters for registry in docker.yaml

    These duplicate the defaults in puppet/services/docker.yaml and
    break things if you include an environment file (e.g that generated
    by quickstart containers-default-parameters.yaml) before the
    docker.yaml.

    Instead it's probably more helpful to include the commented lines
    showing how to enable use of a local docker registry.

    Change-Id: I3896fa2ea7caa603186f0af04f6d8382d50dd97a
    Closes-Bug: #1691524

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 7.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/475277

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/475277
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=6d44f7245ceec4b23145298702bffde1e6e21c1d
Submitter: Jenkins
Branch: master

commit 6d44f7245ceec4b23145298702bffde1e6e21c1d
Author: Martin André <email address hidden>
Date: Mon Jun 19 09:47:31 2017 +0200

    Comment parameters for registry in docker tls env

    This commit brings change from
    I3896fa2ea7caa603186f0af04f6d8382d50dd97a to
    docker-services-tls-everywhere.yaml, which original commit message was:

    These duplicate the defaults in puppet/services/docker.yaml and
    break things if you include an environment file (e.g that generated
    by quickstart containers-default-parameters.yaml) before the
    docker.yaml.

    Instead it's probably more helpful to include the commented lines
    showing how to enable use of a local docker registry.

    Change-Id: Ifa95ef60bc17bd2638ebb6aebf77a819b28c9f0b
    Related-Bug: #1691524