mirror timeouts in upstream causing undercloud and standalone failures

[wes hayutin]

http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22_attach_volume%5C%22%20AND%20module:%5C%22oslo_messaging.rpc.server%5C%22%20AND%20tags:%5C%22screen-c-bak.txt%5C%22%20AND%20voting:1&from=864000s

or

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22429%20Client%20Error%3A%20Too%20Many%20Requests%20for%20url%3A%5C%22%20AND%20voting%3A1

Query: message:"429 Client Error: Too Many Requests for url:" AND voting:1

Error: 2020-07-27T19:18:57.086126Z | HTTPError: 429 Client Error: Too Many Requests for url: http://mirror.ca-ymq-1.vexxhost.opendev.org:8082/v2/tripleotrain/centos-binary-cron/blobs/sha256:76342b0db11c6b5acf33b9f1cbf10b3d2680fb20967ccd7daa9593a39e9e45c0

Errors can be found in the following logs, like the above different providers:

/home/zuul/undercloud_install.log
/home/zuul/standalone_deploy.log

[wes hayutin (weshayutin)]

total hits: 1149
build_branch
  57% stable/train
  33% master
  8% stable/ussuri
  1% stable/stein
build_change
  30% 741332
  8% 743213
  5% 742287
  4% 741228
  4% 742483
build_hostids
  26% 8fff2b99b52d9369c460de43685c4f39059dc99f7b3eef95ec34e8a5
  7% b4b06d164c2a9f69eee6b6bde4382dfed5682dd44da846c0134dfbc5 47da96835961f850d0ef05089fb470e390e2f029601225ae2fc7a16c
  5% d1d66ea3cd7d86ae99e4ee6b4ccc30ff7e4e1fc3e1bb15ed06a375f4 3b588bf9b46984dcbcac2ef8d1e12fd7812a3c543a913e3d951faef8
  4% 5580c89f426017543dbb351de01fb9103575519b0474fae25a22bd73
  3% 3302d80fd0f10672210f806712e5ca99344a304e7b8dfeadc8319d20
build_name
  26% tripleo-ci-centos-7-standalone-upgrade-train
  9% tripleo-ci-centos-7-scenario007-multinode-oooq-container
  6% tripleo-ci-centos-8-containers-undercloud-minion
  6% tripleo-ci-centos-7-standalone
  5% tripleo-ci-centos-8-containerized-undercloud-upgrades
build_node
  52% centos-7
  47% centos-8
build_queue
  84% check
  15% gate
build_status
  100% FAILURE
build_zuul_url
  100% N/A
filename
  100% logs/undercloud/var/log/extra/logstash.txt
log_url
  26% https://dd4052db8565b5d0b0a5-9615cbb5198f5afd96e5019ab78d9def.ssl.cf2.rackcdn.com/741332/3/check/tripleo-ci-centos-7-standalone-upgrade-train/851ea31/logs/undercloud/var/log/extra/logstash.txt
  7% https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_102/743213/1/check/tripleo-ci-centos-7-scenario007-multinode-oooq-container/1028f52/logs/undercloud/var/log/extra/logstash.txt
  5% https://d2c0df89678523ab2148-bae68f4f9acafcdb88c180c1d5a48439.ssl.cf1.rackcdn.com/742287/1/gate/tripleo-ci-centos-8-containers-undercloud-minion/bbba903/logs/undercloud/var/log/extra/logstash.txt
  4% https://69ea250b4e216e013859-23d868cf2a4e074c6e4235ff4566c9f8.ssl.cf1.rackcdn.com/742483/3/gate/tripleo-ci-centos-7-standalone/78a2080/logs/undercloud/var/log/extra/logstash.txt
  3% https://b97e2975f6c80d5d3d55-1c226250a98cd4d99fb41889c8344fe0.ssl.cf1.rackcdn.com/742866/3/check/tripleo-ci-centos-8-containerized-undercloud-upgrades/234e34b/logs/undercloud/var/log/extra/logstash.txt
node_provider
  30% vexxhost-ca-ymq-1
  25% ovh-bhs1
  15% rax-dfw
  13% rax-iad
  9% inap-mtl01
port
  26% 36474
  8% 34044
  5% 51836
  4% 43420
  4% 51092
project
  50% openstack/tripleo-common
  24% openstack/tripleo-heat-templates
  9% openstack/tripleo-ansible
  8% openstack/python-tripleoclient
  4% openstack/tripleo-ci
tags
  100% logstash.txt console postci
voting
  54% 0
  45% 1
zuul_attempts
  96% 1
  3% 2
zuul_executor
  31% ze04.openstack.org
  15% ze03.openstack.org
  13% ze06.openstack.org
  7% ze09.openstack.org
  7% ze12.openstack.org

[wes hayutin (weshayutin)]

[14:33:49] <weshay|ruck> sean-k-mooney, sounds like that may impact mirrors too?
[14:33:50] <sean-k-mooney> the next time a nova patch merges teh promote jobs will run so its not really an issue
[14:34:06] <clarkb> sean-k-mooney: weshay|ruck I think they are separate issues
[14:34:11] <weshay|ruck> k
[14:34:31] <clarkb> sean-k-mooney: your issue was related to afs mounts on the zuul executors not being done before containers started. We've fixed that by restarting the containers and added systemd boot ordering ot have docker start after afs
[14:34:51] <sean-k-mooney> clarkb: cool
[14:35:14] <clarkb> weshay|ruck: your issue is going through the mirrors which run no containers so shouldn't have the same issue
[14:35:28] <sean-k-mooney> i assumed it was a trasiant issue i just got a zull email and noticed it failed.
[14:35:51] <clarkb> weshay|ruck: your issue is dockerhub rate limiting us
[14:35:58] <weshay|ruck> OH NO
[14:36:02] <clarkb> weshay|ruck: last week I think that came up because rdo was doing a million releases or something
[14:36:02] <weshay|ruck> the day has come??
[14:36:18] <weshay|ruck> crap
[14:36:19] <clarkb> and ya'll were going to address that by not trying to consume every single one of them or something
[14:36:39] <clarkb> but the too many requests is coming from docker hub
[14:36:45] <weshay|ruck> ok.. well tbh I'm suprised it took so long
[14:36:49] <weshay|ruck> k
[14:36:54] <clarkb> and generally we don't see it if we're able to use cached layers because we rarely hit docker hub

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/743432

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/743433

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/743629

[Bogdan Dobrelya (bogdando)]

See also https://bugs.launchpad.net/tripleo/+bug/1889372

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/743704

[wes hayutin (weshayutin)]

Also seen in rdo mirrors now

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-master/6973ba4/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: http://mirror.regionone.rdo-cloud-tripleo.rdoproject.org:8082/v2/ceph/daemon/blobs/sha256:ea7eb661de71977e5d57c2d6935d5bca6496780665cc7537595b19f8d431b676
None

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alex Schultz (<email address hidden>) on branch: master
Review: https://review.opendev.org/743629
Reason: meh failed in gate the kolla build

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/743629
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=d3af31414747f28508e236b1fea8db7601fa2e7b
Submitter: Zuul
Branch: master

commit d3af31414747f28508e236b1fea8db7601fa2e7b
Author: Alex Schultz <email address hidden>
Date: Tue Jul 28 14:42:59 2020 -0600

    Handle redirects for blobs better

    This patch adds a new function that checks if a response was a redirect
    for an a request and removes the Authorization header that we usually
    send if it is not one of our trusted hosts. This prevents authorization
    keys from going to insecure places. This is similar logic that exists in
    the moby registry code[0].

    Additionally improves the cachability of blobs from docker.io because
    they are redirects to files that exist on a CDN that doesn't actually
    require authentication. The upstream CI registry caching system doesn't
    cache any requests with the Authorization header per the apache cache
    documentation[1].

    [0] https://github.com/moby/moby/blob/a072d726c753c3b02232e6a7b08d7e7ce79fffa5/registry/registry.go#L140-L174
    [1] https://httpd.apache.org/docs/2.4/caching.html

    Change-Id: I415eec5d307ac73456aa556db9d61ceac1eaa565
    Partial-Bug: #1889122

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/744466

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/744467

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/743433
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=bde0d0084ce37a4e66f74e1c2164eaea06cb3b51
Submitter: Zuul
Branch: master

commit bde0d0084ce37a4e66f74e1c2164eaea06cb3b51
Author: Wes Hayutin <email address hidden>
Date: Mon Jul 27 16:38:45 2020 -0600

    Add native variable to locally build containers

    atm there is an option to turn on local container
    builds for standalone that is defined in zuul only.
    ATM we are considering turning this on for all standalone
    master jobs to reduce the load on docker.io

    This change updates the release vars to to switch it
    on for master. By default in standalone role master
    and ussuri will build locally.

    Depends-On: https://review.opendev.org/#/c/743691/

    Partial-Bug: #1889122
    Change-Id: I32793b94525ece1e64f7924f3f29a7dc2655a588

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/744466
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=aa4825cb53fc4d0b0740f4b4834cf702fcd19f66
Submitter: Zuul
Branch: stable/ussuri

commit aa4825cb53fc4d0b0740f4b4834cf702fcd19f66
Author: Alex Schultz <email address hidden>
Date: Tue Jul 28 14:42:59 2020 -0600

    Handle redirects for blobs better

    This patch adds a new function that checks if a response was a redirect
    for an a request and removes the Authorization header that we usually
    send if it is not one of our trusted hosts. This prevents authorization
    keys from going to insecure places. This is similar logic that exists in
    the moby registry code[0].

    Additionally improves the cachability of blobs from docker.io because
    they are redirects to files that exist on a CDN that doesn't actually
    require authentication. The upstream CI registry caching system doesn't
    cache any requests with the Authorization header per the apache cache
    documentation[1].

    [0] https://github.com/moby/moby/blob/a072d726c753c3b02232e6a7b08d7e7ce79fffa5/registry/registry.go#L140-L174
    [1] https://httpd.apache.org/docs/2.4/caching.html

    Change-Id: I415eec5d307ac73456aa556db9d61ceac1eaa565
    Partial-Bug: #1889122
    (cherry picked from commit d3af31414747f28508e236b1fea8db7601fa2e7b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/744467
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=4dd1bd199f45cbc905924f2936733678a07d4255
Submitter: Zuul
Branch: stable/train

commit 4dd1bd199f45cbc905924f2936733678a07d4255
Author: Alex Schultz <email address hidden>
Date: Tue Jul 28 14:42:59 2020 -0600

    Handle redirects for blobs better

    This patch adds a new function that checks if a response was a redirect
    for an a request and removes the Authorization header that we usually
    send if it is not one of our trusted hosts. This prevents authorization
    keys from going to insecure places. This is similar logic that exists in
    the moby registry code[0].

    Additionally improves the cachability of blobs from docker.io because
    they are redirects to files that exist on a CDN that doesn't actually
    require authentication. The upstream CI registry caching system doesn't
    cache any requests with the Authorization header per the apache cache
    documentation[1].

    [0] https://github.com/moby/moby/blob/a072d726c753c3b02232e6a7b08d7e7ce79fffa5/registry/registry.go#L140-L174
    [1] https://httpd.apache.org/docs/2.4/caching.html

    Change-Id: I415eec5d307ac73456aa556db9d61ceac1eaa565
    Partial-Bug: #1889122
    (cherry picked from commit d3af31414747f28508e236b1fea8db7601fa2e7b)

[Marios Andreou (marios-b)]

still seeing it in gate today at least once there
        * http://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d11/743260/3/gate/tripleo-ci-centos-7-standalone-upgrade-train/d11c9ef/logs/undercloud/home/zuul/standalone_deploy.log
            * HTTPError: 429 Client Error: Too Many Requests for url: http://mirror.dfw.rax.opendev.org:8082/v2/tripleostein/centos-binary-cinder-api/blobs/sha256:399cbc81a4d880cdb2437dd232653d84c93dc6d444ef7e66f4b2ce7ea0d6e47c

[Bogdan Dobrelya (bogdando)]

https://review.opendev.org/743704 is needed as well I guess

[Bogdan Dobrelya (bogdando)]

Hm, the given logs show that "Error detected in auth headers: error invalid_token" happens too often thus ends up rate limited. This is another issue caused by https://review.opendev.org/#/c/743629/

That needs to be also improved, to produce less retries.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/744913

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/744918

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/744952

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/744953

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/744954

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/744955

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/744913
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=5602dfd986939d4d2d19a124603c86504eb32b99
Submitter: Zuul
Branch: master

commit 5602dfd986939d4d2d19a124603c86504eb32b99
Author: Alex Schultz <email address hidden>
Date: Wed Aug 5 08:27:53 2020 -0600

    Reduce workers for label lookups

    Using 16 causes excessive auth requests and gets us rate limited
    upstream. Let's make this dynamic so that the upstream VMs will use less
    workers.

    Partial-Bug: #1889122
    Change-Id: Ib6fed59337f3dac07264052aaf2973ec5a69504f

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/744918
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=014d985f0836340b03d3c2cdeaae8717cfd1ea6b
Submitter: Zuul
Branch: master

commit 014d985f0836340b03d3c2cdeaae8717cfd1ea6b
Author: Alex Schultz <email address hidden>
Date: Wed Aug 5 08:42:17 2020 -0600

    Stop doing tag_from_label lookup with tag

    When using specifying ContainerImagePrepare if a tag is explicitly
    provided in a set, the tag_from_label functionality will not be run as
    we use the defined tag for the containers. Previously we would still
    attempt tag lookups even if we wanted a specific tag.

    This would also cause failures during the deployment if using a
    namespace that didn't have the defined tag_from_label format but we
    defined a specific tag we wanted to use. e.g. using a tag with a md5 vs
    a tag_from_label with numbers in it.

    Change-Id: I4966641aed1a21be60d915ea58dda78b80fe0e1f
    Partial-Bug: #1889122

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/744954
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=933b52b1bd5178dd9fa1229b4b0175a2da911812
Submitter: Zuul
Branch: stable/train

commit 933b52b1bd5178dd9fa1229b4b0175a2da911812
Author: Alex Schultz <email address hidden>
Date: Wed Aug 5 08:27:53 2020 -0600

    Reduce workers for label lookups

    Using 16 causes excessive auth requests and gets us rate limited
    upstream. Let's make this dynamic so that the upstream VMs will use less
    workers.

    Partial-Bug: #1889122
    Change-Id: Ib6fed59337f3dac07264052aaf2973ec5a69504f
    (cherry picked from commit 5602dfd986939d4d2d19a124603c86504eb32b99)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/744952
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=84fc9430e261b41237fc92e58fc41aaea033ce8f
Submitter: Zuul
Branch: stable/ussuri

commit 84fc9430e261b41237fc92e58fc41aaea033ce8f
Author: Alex Schultz <email address hidden>
Date: Wed Aug 5 08:27:53 2020 -0600

    Reduce workers for label lookups

    Using 16 causes excessive auth requests and gets us rate limited
    upstream. Let's make this dynamic so that the upstream VMs will use less
    workers.

    Partial-Bug: #1889122
    Change-Id: Ib6fed59337f3dac07264052aaf2973ec5a69504f
    (cherry picked from commit 5602dfd986939d4d2d19a124603c86504eb32b99)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/744953
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=d3c11cd581dfcd88b86e1a1d17c0c9826fc5676b
Submitter: Zuul
Branch: stable/ussuri

commit d3c11cd581dfcd88b86e1a1d17c0c9826fc5676b
Author: Alex Schultz <email address hidden>
Date: Wed Aug 5 08:42:17 2020 -0600

    Stop doing tag_from_label lookup with tag

    When using specifying ContainerImagePrepare if a tag is explicitly
    provided in a set, the tag_from_label functionality will not be run as
    we use the defined tag for the containers. Previously we would still
    attempt tag lookups even if we wanted a specific tag.

    This would also cause failures during the deployment if using a
    namespace that didn't have the defined tag_from_label format but we
    defined a specific tag we wanted to use. e.g. using a tag with a md5 vs
    a tag_from_label with numbers in it.

    Change-Id: I4966641aed1a21be60d915ea58dda78b80fe0e1f
    Partial-Bug: #1889122
    (cherry picked from commit 014d985f0836340b03d3c2cdeaae8717cfd1ea6b)

[Marios Andreou (marios-b)]

from train gate today (train cherrypick didn't merge yet https://review.opendev.org/#/c/744955)

        * https://66044ceaa9fa4a1fa395-906744ae5f779cf07a1bcffe2431d77c.ssl.cf2.rackcdn.com/741443/6/gate/tripleo-ci-centos-8-undercloud-containers/304105a/logs/undercloud/home/zuul/undercloud_install.log
        * Too Many Requests. Please see https://docs.docker.com/docker-hub/download-rate-limit/

        * https://8f60a2f82243fd880d0e-7daaa7daeb1030fb2122101d5d422feb.ssl.cf1.rackcdn.com/741443/6/gate/tripleo-ci-centos-7-containers-multinode/9daad7a/logs/undercloud/home/zuul/undercloud_install.log
        * Error: Unauthorized for url: http://mirror.ord.rax.opendev.org:8082/v2/tripleotrain/centos-binary-placement-api/blobs/sha256:22839ed824629cb0df6629d91928ff649041131a539f1e54e1d9e50d8db13a0e\nERROR:tripleo_common.image.image_export:[tripleotrain/centos-binary-placement-api] Broken layer found and removed /var/lib/image-serve

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/744955
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=fff6f888a07316c4b2cd612098588a6ebd58a922
Submitter: Zuul
Branch: stable/train

commit fff6f888a07316c4b2cd612098588a6ebd58a922
Author: Alex Schultz <email address hidden>
Date: Wed Aug 5 08:42:17 2020 -0600

    Stop doing tag_from_label lookup with tag

    When using specifying ContainerImagePrepare if a tag is explicitly
    provided in a set, the tag_from_label functionality will not be run as
    we use the defined tag for the containers. Previously we would still
    attempt tag lookups even if we wanted a specific tag.

    This would also cause failures during the deployment if using a
    namespace that didn't have the defined tag_from_label format but we
    defined a specific tag we wanted to use. e.g. using a tag with a md5 vs
    a tag_from_label with numbers in it.

    Change-Id: I4966641aed1a21be60d915ea58dda78b80fe0e1f
    Partial-Bug: #1889122
    (cherry picked from commit 014d985f0836340b03d3c2cdeaae8717cfd1ea6b)

[Bogdan Dobrelya (bogdando)]

This time (see the comment #31) that's not related to _inspect'ing tags but _copy_registry_to_registry:

"ERROR:root:Image prepare failed: 429 Client Error: Too Many Requests for url: http://mirror.ord.rax.opendev.org:8082/v2/tripleotraincentos8/centos-binary-nova-compute-ironic/bl
obs/sha256:0dfe9dc8d91b110ebb357a0f153ab2d826024c34910f10c52fdf1169bd98b55f",
"concurrent.futures.process._RemoteTraceback: ",
"\"\"\"",
"Traceback (most recent call last):",
" File \"/usr/lib64/python3.6/concurrent/futures/process.py\", line 175, in _process_worker",
" r = call_item.fn(*call_item.args, **call_item.kwargs)",
" File \"/usr/lib64/python3.6/concurrent/futures/process.py\", line 153, in _process_chunk",
" return [fn(*args) for args in chunk]",
" File \"/usr/lib64/python3.6/concurrent/futures/process.py\", line 153, in <listcomp>",
" return [fn(*args) for args in chunk]",
" File \"/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py\", line 2559, in upload_task",
" return uploader.upload_image(task)",
" File \"/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py\", line 1545, in upload_image",
" multi_arch=t.multi_arch",
" File \"/usr/lib/python3.6/site-packages/tenacity/__init__.py\", line 292, in wrapped_f",
" return self.call(f, *args, **kw)",
" File \"/usr/lib/python3.6/site-packages/tenacity/__init__.py\", line 358, in call",
" do = self.iter(retry_state=retry_state)",
" File \"/usr/lib/python3.6/site-packages/tenacity/__init__.py\", line 331, in iter",
" raise retry_exc.reraise()",
" File \"/usr/lib/python3.6/site-packages/tenacity/__init__.py\", line 167, in reraise",
" raise self.last_attempt.result()",
" File \"/usr/lib64/python3.6/concurrent/futures/_base.py\", line 425, in result",
" return self.__get_result()",
" File \"/usr/lib64/python3.6/concurrent/futures/_base.py\", line 384, in __get_result",
" raise self._exception",
" File \"/usr/lib/python3.6/site-packages/tenacity/__init__.py\", line 361, in call",
" result = fn(*args, **kwargs)",
" File \"/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py\", line 1920, in _copy_registry_to_registry",

So the means that https://review.opendev.org/744955 will not help here

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/745170

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/745170
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=d4f6eb64d4a54cfa90d4c1572ef54495f151fea5
Submitter: Zuul
Branch: master

commit d4f6eb64d4a54cfa90d4c1572ef54495f151fea5
Author: Alex Schultz <email address hidden>
Date: Thu Aug 6 09:21:59 2020 -0600

    Add exponential backoff to ratelimited requests

    If we're ratelimited on a request, we should retry the request but back
    off significantly. Previously we're not really handling 429 any
    different than other requests which may lead to the requests being
    retried too quickly.

    Change-Id: I3832332abdfd7daaf373dc0924fec268f159d774
    Related-Bug: #1889372
    Related-bug: #1889122

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/745322

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/745323

[OpenStack Infra (hudson-openstack)]

Change abandoned by Bogdan Dobrelya (bogdando) (<email address hidden>) on branch: master
Review: https://review.opendev.org/743704

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/745322
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=be81e6ee690a8a5f8bc15309d333ea8266f43af2
Submitter: Zuul
Branch: stable/ussuri

commit be81e6ee690a8a5f8bc15309d333ea8266f43af2
Author: Alex Schultz <email address hidden>
Date: Thu Aug 6 09:21:59 2020 -0600

    Add exponential backoff to ratelimited requests

    If we're ratelimited on a request, we should retry the request but back
    off significantly. Previously we're not really handling 429 any
    different than other requests which may lead to the requests being
    retried too quickly.

    Change-Id: I3832332abdfd7daaf373dc0924fec268f159d774
    Related-Bug: #1889372
    Related-bug: #1889122
    (cherry picked from commit d4f6eb64d4a54cfa90d4c1572ef54495f151fea5)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/745323
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=8b379fb9f0528a1b7b034d1a3f5e1fcfe64201ef
Submitter: Zuul
Branch: stable/train

commit 8b379fb9f0528a1b7b034d1a3f5e1fcfe64201ef
Author: Alex Schultz <email address hidden>
Date: Thu Aug 6 09:21:59 2020 -0600

    Add exponential backoff to ratelimited requests

    If we're ratelimited on a request, we should retry the request but back
    off significantly. Previously we're not really handling 429 any
    different than other requests which may lead to the requests being
    retried too quickly.

    Change-Id: I3832332abdfd7daaf373dc0924fec268f159d774
    Related-Bug: #1889372
    Related-bug: #1889122
    (cherry picked from commit d4f6eb64d4a54cfa90d4c1572ef54495f151fea5)

[Bogdan Dobrelya (bogdando)]

It is also addressed by https://review.opendev.org/#/c/745718

[Bogdan Dobrelya (bogdando)]

It seems the issue is back, see example log

https://efcd20f9fe9adc863f59-2d35d76e65f9afb322cf96e46e6932b6.ssl.cf1.rackcdn.com/745825/3/check/tripleo-ci-centos-8-scenario004-standalone/6165086/logs/undercloud/home/zuul/standalone_deploy.log

[Bogdan Dobrelya (bogdando)]

The slightly adjusted logstash query http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Rate%20Limited%20while%5C%22%20AND%20voting:1 shows some hits for the past 7 days

[Marios Andreou (marios-b)]

@bogdan yes agree i saw one yesterday but forgot to update the bug (meant to move it back to triaged)

https://7ca43678c3b20511f0b0-d73e4d97d0ba955d901e72d18343bae6.ssl.cf1.rackcdn.com/746592/1/check/tripleo-ci-centos-8-containers-multinode-ussuri/551811c/logs/undercloud/home/zuul/overcloud_deploy.log

2020-08-17 23:07:44 | 2020-08-17 23:07:44.494974 | bc764e04-6c4a-92a7-7ec7-00000000000e | FATAL | Prepare Container images and update plan | localhost | error={"changed": false, "error": "Rate Limited while requesting http://mirror.dfw.rax.opendev.org:8082/v2/tripleou/centos-binary-cron/blobs/sha256:8f2fb13f44b99a094205ca475d744a473270c0a7c337248f92216aca79b2acc7", "msg": "Error updating image parms for plan overcloud: Rate Limited while requesting http://mirror.dfw.rax.opendev.org:8082/v2/tripleou/centos-binary-cron/blobs/sha256:8f2fb13f44b99a094205ca475d744a473270c0a7c337248f92216aca79b2acc7", "success": false}

I also came across at last one today there
        * http://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_112/746724/1/check/tripleo-ci-centos-8-containers-multinode/112f00a/logs/undercloud/home/zuul/undercloud_install.log
        * Exception: Rate Limited while requesting http://mirror.iad.rax.opendev.org:8082/v2/tripleomaster/centos-binary-cron/blobs/sha256:29ac218f443ea6cbdb1b1201c4d025ee7004c43671c85b05082a26a521b945d3

[Bogdan Dobrelya (bogdando)]

We need https://review.opendev.org/#/c/745492/ to get logs back

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/746990

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/746993

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/747416

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/746990
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=f4d312f18633c6142964075f74ec7afe582a5c3e
Submitter: Zuul
Branch: master

commit f4d312f18633c6142964075f74ec7afe582a5c3e
Author: Alex Schultz <email address hidden>
Date: Wed Aug 19 10:30:11 2020 -0600

    Add modify_only_with_source

    Currently the modify_only_with_labels has to query the container
    registry to see what containers we should modify. We can use the source
    in from our service to container mapping to limit which containers we
    want to update. In the upstream CI, we only want to modify the kolla or
    tripleo containers as we want to prevent ceph and other related
    containers from being updated.

    Change-Id: I4bff2b96f7b13bde808f929c3567dcf167f1eacd
    Related-Bug: #1889122

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/747416
Committed: https://git.openstack.org/cgit/openstack/ansible-role-collect-logs/commit/?id=b411f774b569a76ac54747b5fbe4d0d35f00a061
Submitter: Zuul
Branch: master

commit b411f774b569a76ac54747b5fbe4d0d35f00a061
Author: Wes Hayutin <email address hidden>
Date: Fri Aug 21 07:40:49 2020 -0600

    add sova pattern for new tc exception, ratelimit

    capture errors like..
    tripleo_common.image.exception.ImageRateLimitedException:
    Rate Limited while requesting http://mirror.iad.rax.open
    dev.org:8082/v2/tripleotraincentos8/centos-binary-barbi
    can-api/blobs/sha256:debac393b2f14

    Partial-Bug: #1889122
    Change-Id: Ie444f4e12129a16934afa58d6ae65d16bdfedfae

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/747720

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/747721

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/747720
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=0f67579c4db03731907e28d9d7a28e662ef6d9f7
Submitter: Zuul
Branch: stable/ussuri

commit 0f67579c4db03731907e28d9d7a28e662ef6d9f7
Author: Alex Schultz <email address hidden>
Date: Wed Aug 19 10:30:11 2020 -0600

    Add modify_only_with_source

    Currently the modify_only_with_labels has to query the container
    registry to see what containers we should modify. We can use the source
    in from our service to container mapping to limit which containers we
    want to update. In the upstream CI, we only want to modify the kolla or
    tripleo containers as we want to prevent ceph and other related
    containers from being updated.

    Change-Id: I4bff2b96f7b13bde808f929c3567dcf167f1eacd
    Related-Bug: #1889122
    (cherry picked from commit f4d312f18633c6142964075f74ec7afe582a5c3e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/747721
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=78ed3b003901fa7a2df6aba1cc519295a7492e8c
Submitter: Zuul
Branch: stable/train

commit 78ed3b003901fa7a2df6aba1cc519295a7492e8c
Author: Alex Schultz <email address hidden>
Date: Wed Aug 19 10:30:11 2020 -0600

    Add modify_only_with_source

    Currently the modify_only_with_labels has to query the container
    registry to see what containers we should modify. We can use the source
    in from our service to container mapping to limit which containers we
    want to update. In the upstream CI, we only want to modify the kolla or
    tripleo containers as we want to prevent ceph and other related
    containers from being updated.

    Change-Id: I4bff2b96f7b13bde808f929c3567dcf167f1eacd
    Related-Bug: #1889122
    (cherry picked from commit f4d312f18633c6142964075f74ec7afe582a5c3e)

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alex Schultz (<email address hidden>) on branch: master
Review: https://review.opendev.org/746993

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/746993
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=161099faa3be7130b7843dd72674742e889ca0d7
Submitter: Zuul
Branch: master

commit 161099faa3be7130b7843dd72674742e889ca0d7
Author: Alex Schultz <email address hidden>
Date: Wed Aug 19 10:36:47 2020 -0600

    Use new modify_only_with_source

    The modify_only_with_labels has to query the container registry in order
    to do updates. In the upstream CI we want to limit the containers that
    are to be updated to only if the image_source is 'kolla' or 'tripleo'.

    Change-Id: I9ac9ef61097443a00b39710f122bf147d4012e4c
    Depends-On: https://review.opendev.org/#/c/746990
    Depends-On: https://review.opendev.org/#/c/747721
    Depends-On: https://review.opendev.org/#/c/747720
    Related-Bug: #1889122

[OpenStack Infra (hudson-openstack)]

Change abandoned by wes hayutin (<email address hidden>) on branch: master
Review: https://review.opendev.org/743432
Reason: https://review.opendev.org/754202

[Ronelle Landy (rlandy)]

closing this out