Node exporter container fails to push to registry with 500 error and "invalid checksum digest format"

Bug #1908844 reported by Sagi (Sergey) Shnaidman
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Unassigned

Bug Description

Content provider job sometimes fail on trying to push node-exporter container to local registry(?):

https://763d37e4466a2871dd4c-4cebd80e32fd7e7586df4103c9478cce.ssl.cf1.rackcdn.com/767321/1/gate/tripleo-ci-centos-8-content-provider/c7d7ce4/job-output.txt

error pushing image "quay.io/prometheus/node-exporter:v0.17.0" to "docker://127.0.0.1:5001/tripleotraincentos8/node-exporter:v0.17.0": error copying layers and metadata from "containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage]quay.io/prometheus/node-exporter:v0.17.0" to "docker://127.0.0.1:5001/tripleotraincentos8/node-exporter:v0.17.0": Error trying to reuse blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 at destination: failed to read from destination repository tripleotraincentos8/node-exporter: 500 (Internal Server Error)

https://085a1f26fa6edfbd4a66-e4b92f5e1fb788c5da82bf1447ffa549.ssl.cf1.rackcdn.com/764638/8/check/tripleo-ci-centos-8-content-provider/67727bd/job-output.txt

error pushing image "quay.io/prometheus/node-exporter:v0.17.0" to "docker://127.0.0.1:5001/tripleomaster/node-exporter:v0.17.0": error copying layers and metadata from "containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage]quay.io/prometheus/node-exporter:v0.17.0" to "docker://127.0.0.1:5001/tripleomaster/node-exporter:v0.17.0": Error trying to reuse blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 at destination: failed to read from destination repository tripleomaster/node-exporter: 500 (Internal Server Error)

From stdout of registry there is more detailed error: invalid checksum digest format

https://763d37e4466a2871dd4c-4cebd80e32fd7e7586df4103c9478cce.ssl.cf1.rackcdn.com/767321/1/gate/tripleo-ci-centos-8-content-provider/c7d7ce4/logs/undercloud/var/log/extra/podman/containers/docker_registry/stdout.log

time="2020-12-17T09:15:40.466406675Z" level=error msg="response completed with error" err.code=unknown err.detail="invalid checksum digest format" err.message="unknown error" go.version=go1.11.2 http.request.host="127.0.0.1:5001" http.request.id=a856451c-f413-427f-b36d-d35fe5ba2f83 http.request.method=HEAD http.request.remoteaddr="10.88.0.1:49640" http.request.uri="/v2/tripleotraincentos8/node-exporter/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" http.request.useragent="Buildah/1.11.6" http.response.contenttype="application/json; charset=utf-8" http.response.duration=4.470206ms http.response.status=500 http.response.written=70 vars.digest="sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" vars.name="tripleotraincentos8/node-exporter"
10.88.0.1 - - [17/Dec/2020:09:15:40 +0000] "HEAD /v2/tripleotraincentos8/node-exporter/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 HTTP/1.1" 500 70 "" "Buildah/1.11.6"

summary: - Node exporter container fails to push to registry with 500 error
+ Node exporter container fails to push to registry with 500 error and
+ "invalid checksum digest format"
Revision history for this message
Sagi (Sergey) Shnaidman (sshnaidm) wrote :

Recent example:
https://d3ba52c45612db77e201-20f8b612e9341a4bfb8242ee0493df0d.ssl.cf2.rackcdn.com/769481/1/check/tripleo-ci-centos-8-content-provider-ussuri/d7fc558/job-output.txt

error pushing image "quay.io/prometheus/node-exporter:v0.17.0" to "docker://127.0.0.1:5001/tripleou/node-exporter:v0.17.0": error copying layers and metadata from "containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage]quay.io/prometheus/node-exporter:v0.17.0" to "docker://127.0.0.1:5001/tripleou/node-exporter:v0.17.0": Error trying to reuse blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 at destination: failed to read from destination repository tripleou/node-exporter: 500 (Internal Server Error)

https://d3ba52c45612db77e201-20f8b612e9341a4bfb8242ee0493df0d.ssl.cf2.rackcdn.com/769481/1/check/tripleo-ci-centos-8-content-provider-ussuri/d7fc558/logs/undercloud/var/log/extra/podman/containers/docker_registry/stdout.log

time="2021-01-06T10:24:22.25221758Z" level=error msg="response completed with error" err.code=unknown err.detail="invalid checksum digest format" err.message="unknown error" go.version=go1.11.2 http.request.host="127.0.0.1:5001" http.request.id=1a02776f-9842-4bea-9d7a-0a44719e5699 http.request.method=HEAD http.request.remoteaddr="10.88.0.1:58718" http.request.uri="/v2/tripleou/node-exporter/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" http.request.useragent="Buildah/1.11.6" http.response.contenttype="application/json; charset=utf-8" http.response.duration=8.229976ms http.response.status=500 http.response.written=70 vars.digest="sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" vars.name="tripleou/node-exporter"
10.88.0.1 - - [06/Jan/2021:10:24:22 +0000] "HEAD /v2/tripleou/node-exporter/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 HTTP/1.1" 500 70 "" "Buildah/1.11.6"

wes hayutin (weshayutin)
tags: added: promotion-blocker
Revision history for this message
wes hayutin (weshayutin) wrote :
Revision history for this message
Sagi (Sergey) Shnaidman (sshnaidm) wrote :
Revision history for this message
Sagi (Sergey) Shnaidman (sshnaidm) wrote :
Revision history for this message
Sagi (Sergey) Shnaidman (sshnaidm) wrote :
Changed in tripleo:
milestone: wallaby-2 → wallaby-3
Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers