Content provider job sometimes fail on trying to push node-exporter container to local registry(?):
https://763d37e4466a2871dd4c-4cebd80e32fd7e7586df4103c9478cce.ssl.cf1.rackcdn.com/767321/1/gate/tripleo-ci-centos-8-content-provider/c7d7ce4/job-output.txt
error pushing image "quay.io/prometheus/node-exporter:v0.17.0" to "docker://127.0.0.1:5001/tripleotraincentos8/node-exporter:v0.17.0": error copying layers and metadata from "containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage]quay.io/prometheus/node-exporter:v0.17.0" to "docker://127.0.0.1:5001/tripleotraincentos8/node-exporter:v0.17.0": Error trying to reuse blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 at destination: failed to read from destination repository tripleotraincentos8/node-exporter: 500 (Internal Server Error)
https://085a1f26fa6edfbd4a66-e4b92f5e1fb788c5da82bf1447ffa549.ssl.cf1.rackcdn.com/764638/8/check/tripleo-ci-centos-8-content-provider/67727bd/job-output.txt
error pushing image "quay.io/prometheus/node-exporter:v0.17.0" to "docker://127.0.0.1:5001/tripleomaster/node-exporter:v0.17.0": error copying layers and metadata from "containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage]quay.io/prometheus/node-exporter:v0.17.0" to "docker://127.0.0.1:5001/tripleomaster/node-exporter:v0.17.0": Error trying to reuse blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 at destination: failed to read from destination repository tripleomaster/node-exporter: 500 (Internal Server Error)
From stdout of registry there is more detailed error: invalid checksum digest format
https://763d37e4466a2871dd4c-4cebd80e32fd7e7586df4103c9478cce.ssl.cf1.rackcdn.com/767321/1/gate/tripleo-ci-centos-8-content-provider/c7d7ce4/logs/undercloud/var/log/extra/podman/containers/docker_registry/stdout.log
time="2020-12-17T09:15:40.466406675Z" level=error msg="response completed with error" err.code=unknown err.detail="invalid checksum digest format" err.message="unknown error" go.version=go1.11.2 http.request.host="127.0.0.1:5001" http.request.id=a856451c-f413-427f-b36d-d35fe5ba2f83 http.request.method=HEAD http.request.remoteaddr="10.88.0.1:49640" http.request.uri="/v2/tripleotraincentos8/node-exporter/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" http.request.useragent="Buildah/1.11.6" http.response.contenttype="application/json; charset=utf-8" http.response.duration=4.470206ms http.response.status=500 http.response.written=70 vars.digest="sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" vars.name="tripleotraincentos8/node-exporter"
10.88.0.1 - - [17/Dec/2020:09:15:40 +0000] "HEAD /v2/tripleotraincentos8/node-exporter/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 HTTP/1.1" 500 70 "" "Buildah/1.11.6"
Recent example: /d3ba52c45612db 77e201- 20f8b612e9341a4 bfb8242ee0493df 0d.ssl. cf2.rackcdn. com/769481/ 1/check/ tripleo- ci-centos- 8-content- provider- ussuri/ d7fc558/ job-output. txt
https:/
error pushing image "quay.io/ prometheus/ node-exporter: v0.17.0" to "docker: //127.0. 0.1:5001/ tripleou/ node-exporter: v0.17.0" : error copying layers and metadata from "containers- storage: [overlay@ /var/lib/ containers/ storage+ /var/run/ containers/ storage] quay.io/ prometheus/ node-exporter: v0.17.0" to "docker: //127.0. 0.1:5001/ tripleou/ node-exporter: v0.17.0" : Error trying to reuse blob sha256: a3ed95caeb02ffe 68cdd9fd8440668 0ae93d633cb1642 2d00e8a7c22955b 46d4 at destination: failed to read from destination repository tripleou/ node-exporter: 500 (Internal Server Error)
https:/ /d3ba52c45612db 77e201- 20f8b612e9341a4 bfb8242ee0493df 0d.ssl. cf2.rackcdn. com/769481/ 1/check/ tripleo- ci-centos- 8-content- provider- ussuri/ d7fc558/ logs/undercloud /var/log/ extra/podman/ containers/ docker_ registry/ stdout. log
time="2021- 01-06T10: 24:22.25221758Z " level=error msg="response completed with error" err.code=unknown err.detail="invalid checksum digest format" err.message= "unknown error" go.version=go1.11.2 http.request. host="127. 0.0.1:5001" http.request. id=1a02776f- 9842-4bea- 9d7a-0a44719e56 99 http.request. method= HEAD http.request. remoteaddr= "10.88. 0.1:58718" http.request. uri="/v2/ tripleou/ node-exporter/ blobs/sha256: a3ed95caeb02ffe 68cdd9fd8440668 0ae93d633cb1642 2d00e8a7c22955b 46d4" http.request. useragent= "Buildah/ 1.11.6" http.response. contenttype= "application/ json; charset=utf-8" http.response. duration= 8.229976ms http.response. status= 500 http.response. written= 70 vars.digest= "sha256: a3ed95caeb02ffe 68cdd9fd8440668 0ae93d633cb1642 2d00e8a7c22955b 46d4" vars.name= "tripleou/ node-exporter" 2021:10: 24:22 +0000] "HEAD /v2/tripleou/ node-exporter/ blobs/sha256: a3ed95caeb02ffe 68cdd9fd8440668 0ae93d633cb1642 2d00e8a7c22955b 46d4 HTTP/1.1" 500 70 "" "Buildah/1.11.6"
10.88.0.1 - - [06/Jan/