Node exporter container fails to push to registry with 500 error and "invalid checksum digest format"

[Sagi (Sergey) Shnaidman]

Content provider job sometimes fail on trying to push node-exporter container to local registry(?):

https://763d37e4466a2871dd4c-4cebd80e32fd7e7586df4103c9478cce.ssl.cf1.rackcdn.com/767321/1/gate/tripleo-ci-centos-8-content-provider/c7d7ce4/job-output.txt

error pushing image "quay.io/prometheus/node-exporter:v0.17.0" to "docker://127.0.0.1:5001/tripleotraincentos8/node-exporter:v0.17.0": error copying layers and metadata from "containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage]quay.io/prometheus/node-exporter:v0.17.0" to "docker://127.0.0.1:5001/tripleotraincentos8/node-exporter:v0.17.0": Error trying to reuse blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 at destination: failed to read from destination repository tripleotraincentos8/node-exporter: 500 (Internal Server Error)

https://085a1f26fa6edfbd4a66-e4b92f5e1fb788c5da82bf1447ffa549.ssl.cf1.rackcdn.com/764638/8/check/tripleo-ci-centos-8-content-provider/67727bd/job-output.txt

error pushing image "quay.io/prometheus/node-exporter:v0.17.0" to "docker://127.0.0.1:5001/tripleomaster/node-exporter:v0.17.0": error copying layers and metadata from "containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage]quay.io/prometheus/node-exporter:v0.17.0" to "docker://127.0.0.1:5001/tripleomaster/node-exporter:v0.17.0": Error trying to reuse blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 at destination: failed to read from destination repository tripleomaster/node-exporter: 500 (Internal Server Error)

From stdout of registry there is more detailed error: invalid checksum digest format

https://763d37e4466a2871dd4c-4cebd80e32fd7e7586df4103c9478cce.ssl.cf1.rackcdn.com/767321/1/gate/tripleo-ci-centos-8-content-provider/c7d7ce4/logs/undercloud/var/log/extra/podman/containers/docker_registry/stdout.log

time="2020-12-17T09:15:40.466406675Z" level=error msg="response completed with error" err.code=unknown err.detail="invalid checksum digest format" err.message="unknown error" go.version=go1.11.2 http.request.host="127.0.0.1:5001" http.request.id=a856451c-f413-427f-b36d-d35fe5ba2f83 http.request.method=HEAD http.request.remoteaddr="10.88.0.1:49640" http.request.uri="/v2/tripleotraincentos8/node-exporter/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" http.request.useragent="Buildah/1.11.6" http.response.contenttype="application/json; charset=utf-8" http.response.duration=4.470206ms http.response.status=500 http.response.written=70 vars.digest="sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" vars.name="tripleotraincentos8/node-exporter"
10.88.0.1 - - [17/Dec/2020:09:15:40 +0000] "HEAD /v2/tripleotraincentos8/node-exporter/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 HTTP/1.1" 500 70 "" "Buildah/1.11.6"

[Sagi (Sergey) Shnaidman (sshnaidm)]

Recent example:
https://d3ba52c45612db77e201-20f8b612e9341a4bfb8242ee0493df0d.ssl.cf2.rackcdn.com/769481/1/check/tripleo-ci-centos-8-content-provider-ussuri/d7fc558/job-output.txt

error pushing image "quay.io/prometheus/node-exporter:v0.17.0" to "docker://127.0.0.1:5001/tripleou/node-exporter:v0.17.0": error copying layers and metadata from "containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage]quay.io/prometheus/node-exporter:v0.17.0" to "docker://127.0.0.1:5001/tripleou/node-exporter:v0.17.0": Error trying to reuse blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 at destination: failed to read from destination repository tripleou/node-exporter: 500 (Internal Server Error)

https://d3ba52c45612db77e201-20f8b612e9341a4bfb8242ee0493df0d.ssl.cf2.rackcdn.com/769481/1/check/tripleo-ci-centos-8-content-provider-ussuri/d7fc558/logs/undercloud/var/log/extra/podman/containers/docker_registry/stdout.log

time="2021-01-06T10:24:22.25221758Z" level=error msg="response completed with error" err.code=unknown err.detail="invalid checksum digest format" err.message="unknown error" go.version=go1.11.2 http.request.host="127.0.0.1:5001" http.request.id=1a02776f-9842-4bea-9d7a-0a44719e5699 http.request.method=HEAD http.request.remoteaddr="10.88.0.1:58718" http.request.uri="/v2/tripleou/node-exporter/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" http.request.useragent="Buildah/1.11.6" http.response.contenttype="application/json; charset=utf-8" http.response.duration=8.229976ms http.response.status=500 http.response.written=70 vars.digest="sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" vars.name="tripleou/node-exporter"
10.88.0.1 - - [06/Jan/2021:10:24:22 +0000] "HEAD /v2/tripleou/node-exporter/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 HTTP/1.1" 500 70 "" "Buildah/1.11.6"

[wes hayutin (weshayutin)]

just hit the gate: here:

https://c2b610900eb657fe6f2e-f4b3e6a1d73b0d4e13079b70a1af7060.ssl.cf1.rackcdn.com/768046/3/gate/tripleo-ci-centos-8-content-provider/f91d657/job-output.txt

[Sagi (Sergey) Shnaidman (sshnaidm)]

Should be resolved by https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/769613

[Sagi (Sergey) Shnaidman (sshnaidm)]

Still see it here:

09 Jan
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b4f/769131/3/check/tripleo-ci-centos-8-content-provider-ussuri/b4fea15/job-output.txt

11 Jan
https://b535908aa116a5454978-8b5cc018a02fed487ca2a6d59a4ee9d5.ssl.cf5.rackcdn.com/769985/2/check/tripleo-ci-centos-8-content-provider-victoria/129e24b/job-output.txt

[Sagi (Sergey) Shnaidman (sshnaidm)]

Should help: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/770303