platform-integ-apps failed to download image due to public registry rate limiting

[Peng Peng]

Brief Description
-----------------
System initialization failed by platform-integ-apps apply failed. sysinv.log shows
 Deployment of application platform-integ-apps (1.0-9) failed: failed to download one or more image(s).

Severity
--------
Critical

Steps to Reproduce
------------------
system initial

TC-name:

Expected Behavior
------------------
application platform-integ-apps apply success

Actual Behavior
----------------
application platform-integ-apps apply failed.

Reproducibility
---------------
Reproducible
2/2

System Configuration
--------------------
One node system
Multi-node system

Lab-name: SM-3 WCP_99-103

Branch/Pull Time/Commit
-----------------------
2020-11-26_00-00-06

Last Pass
---------
2020-11-25_00-00-08

Timestamp/Logs
--------------
 Image download failed: docker.io/starlingx/k8s-cni-sriov:stx.3.0-v2.2500 Server Error: Internal Server Error (\"toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:stx.3.0-v2.2 ...\n

sysinv 2020-11-26 09:38:51.632 100687 ERROR sysinv.conductor.kube_app [-] Image docker.io/starlingx/ceph-config-helper:v1.15.0 download failed from public/privateregistry: 500 Server Error: Internal Server Error ("toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit"): APIError: 500 Server Error: Internal Server Error ("toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit")
sysinv 2020-11-26 09:38:51.632 100687 INFO sysinv.conductor.kube_app [-] Failed to download image: registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0
sysinv 2020-11-26 09:38:51.633 100687 ERROR sysinv.conductor.kube_app [-] Deployment of application platform-integ-apps (1.0-9) failed: failed to download one or more image(s).: KubeAppApplyFailure: Deployment of application platform-integ-apps (1.0-9) failed: failed to download one or more image(s).

Test Activity
-------------
installation

[Peng Peng (ppeng)]

collect log loaded at:
https://files.starlingx.kube.cengn.ca/launchpad/1906298

[Ghada Khalil (gkhalil)]

Docker hub rate limiting took effect on Nov20, 2020.

For details, see: https://www.docker.com/increase-rate-limits?utm_source=docker&utm_medium=web%20referral&utm_campaign=increase%20rate%20limit&utm_budget=

[Ghada Khalil (gkhalil)]

This is not a software issue with StarlingX. It's related to the public dockerhub account used to keep the stx docker images. Bill Zvonar is investigating moving us to a dockerhub pro account. Assigning this LP to him to use for tracking resolution.

[Ghada Khalil (gkhalil)]

Note: This issue is mitigated by using a private registry for the container images, so it should not block any stx development/test activities.