DockerHub limits causing failure despite having a local proxy

[Boris Lukashev]

Large deployments fail consistently with dockerhub rate limit errors (sorta defeats the point of dockerhub imo), despite having a local proxy set up per https://docs.openstack.org/kolla-ansible/latest/user/multinode.html using
```
docker_custom_config:
  registry-mirrors:
    - http://<ip_address>:<mapped_port from container>
```
The error occurs with the proxy running or not running, so i'm guessing that its being ignored at runtime.

Is there a better way to avoid these limits, or instructions somewhere on just building the images in-house to the local registry?

[Radosław Piliszek (yoctozepto)]

Hmm, it would confirm my earlier observation that, while the mirror caches images, it still asks DockerHub about them each time, causing the quota to deplete. Sad.

Anyhow, yes; it's generally recommended for production to entirely control your images, i.e., build them yourself using Kolla (kolla-build command) and publish to a locally-controlled registry.

For the registry I use Harbor: https://goharbor.io/
It's quite popular and works well.

As for building the images: https://docs.openstack.org/kolla/latest/admin/image-building.html

[Radosław Piliszek (yoctozepto)]

I have added this issue for discussion during our internal meeting: https://wiki.openstack.org/wiki/Meetings/Kolla

[Boris Lukashev (rageltman)]

Thank you sir.
Now that Harbor is not a VMWare product, is it feasible to include it in the kolla-ansible stack as a dedicated proxy-cache using their quicksetup appprach sans TLS?

[Radosław Piliszek (yoctozepto)]

We are open to code contributions. It could be done with TLS as well but without will probably be simpler.

[Piotr Parczewski (parczewski)]

Classic on-premise Docker registry is already included if you don't need fancy Harbor features. https://docs.openstack.org/kolla-ansible/latest/user/multinode.html#option-1-local-registry

[Radosław Piliszek (yoctozepto)]

"Included" is a big word. It's merely simpler to use. If one does not need fancy features, often useful in production, then the barebones version is fine as it's enough to run for Kolla Ansible.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/808486

[Radosław Piliszek (yoctozepto)]

ml notified http://lists.openstack.org/pipermail/openstack-discuss/2021-September/024834.html

[Radosław Piliszek (yoctozepto)]

We will default to Quay.io in Xena+. And inform users about the choice on the stable branches.

[Boris Lukashev (rageltman)]

Thank you

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/808486
Committed: https://opendev.org/openstack/kolla-ansible/commit/0d9477de3899886165eae2b3a06ef316bebf59d8
Submitter: "Zuul (22348)"
Branch: master

commit 0d9477de3899886165eae2b3a06ef316bebf59d8
Author: Radosław Piliszek <email address hidden>
Date: Fri Sep 10 18:30:25 2021 +0000

    Switch default images source to quay.io

    Docs adapted to match.
    Removed the unsupported-for-quay option to set up
    a pull-through cache.

    Closes-Bug: #1942134
    Change-Id: If5a26b1ba4bf35bc29306c24f608396dbf5e3371

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 13.0.0.0rc1 release candidate.