shutting down the store failing on build push container job: 500 Internal Server

[Bhagyashri Shewale]

periodic-tripleo-ci-build-containers-ubi-push is failing on - master, wallaby, victoria, ussuri and c8 train with below error:

2021-12-27 00:26:21.502934 | TASK [build-containers : Run image build as ansible user > /home/zuul/workspace/logs/build.log]
2021-12-27 00:32:16.546206 | primary | ERROR
2021-12-27 00:32:16.547429 | primary | {
2021-12-27 00:32:16.547549 | primary | "delta": "0:05:54.324973",
2021-12-27 00:32:16.547619 | primary | "end": "2021-12-27 00:32:16.192991",
2021-12-27 00:32:16.547680 | primary | "msg": "non-zero return code",
2021-12-27 00:32:16.547738 | primary | "rc": 1,
2021-12-27 00:32:16.547796 | primary | "start": "2021-12-27 00:26:21.868018"
2021-12-27 00:32:16.547896 | primary | }
2021-12-27 00:32:16.576636 |
2021-12-27 00:32:16.576793 | PLAY RECAP
2021-12-27 00:32:16.576968 | primary | ok: 23 changed: 9 unreachable: 0 failed: 1 skipped: 29 rescued: 0 ignored: 0
2021-12-27 00:32:16.577056 |

Error initiating layer upload to /v2/tripleomaster/openstack-base/blobs/uploads/ in trunk.registry.rdoproject.org: received unexpected HTTP status: 500 Internal Server Error\nlevel=debug msg="shutting down the store"\n'
2021-12-27 00:32:15 | 2021-12-27 00:32:15.720 42351 ERROR openstack 
2021-12-27 00:32:15 | 2021-12-27 00:32:15.721 42351 DEBUG osc_lib.shell [-] clean_up Build:
2021-12-27 00:32:15 | The following errors were detected during container build(s):
2021-12-27 00:32:15 |
2021-12-27 00:32:15 | Exception information: Unexpected error while running command.
2021-12-27 00:32:15 | Command: sudo buildah --log-level=debug push --tls-verify=False trunk.registry.rdoproject.org/tripleomaster/openstack-base:f527c9f797367c5c38e5f308573cd318 docker://trunk.registry.rdoproject.org/tripleomaster/openstack-base:f527c9f797367c5c38e5f308573cd318
2021-12-27 00:32:15 | Exit code: 125
2021-12-27 00:32:15 | Stdout: ''
2021-12-27 00:32:15 | Stderr: 'level=debug msg="[graphdriver] trying provided driver \\"overlay\\""\nlevel=debug msg="cached value indicated that overlay is supported"\nlevel=debug msg="cached value indicated that metacopy is being used"\nlevel=debug msg="cached value indicated that native-diff is not being used"\nlevel=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled"\nlevel=debug msg="backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true"\nlevel=debug msg="Loading registries configuration \\"/etc/containers/registries.conf\\""\nlevel=debug msg="Loading registries configuration \\"/etc/containers/registries.conf.d/000-shortnames.conf\\""\nlevel=debug msg="Loading registries configuration \\"/etc/containers/registries.conf.d/rhel-shortnames.conf\\""\nlevel=debug msg="registry \\"trunk.registry.rdoproject.org\\" is not listed in registries configuration \\"/etc/containers/registries.conf\\", assuming it\'s not blocked"\nlevel=debug msg="parsed reference into \\"[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]trunk.registry.rdoproject.org/tripleomaster/openstack-base:f527c9f797367c5c38e5f308573cd318\\""\nlevel=debug msg="pushing image to reference \\"docker://trunk.registry.rdoproject.org/tripleomaster/openstack-base:f527c9f797367c5c38e5f308573cd318\\" is allowed by policy"\nlevel=debug msg="Returning credentials from /run/containers/0/auth.json"\nlevel=debug msg="Using registries.d directory /etc/containers/registries.d for sigstore configuration"\nlevel=debug msg=" Using \\"default-docker\\" configuration"\nlevel=debug msg=" Using file:///var/lib/containers/sigstore"\nlevel=debug msg="Looking for TLS certificates and private keys in /etc/docker/certs.d/trunk.registry.rdoproject.org"\nlevel=debug msg="Using blob info cache at /var/lib/containers/cache/blob-info-cache-v1.boltdb"\nlevel=debug msg="IsRunningImageAllowed for image containers-storage:[overlay@/var/lib/containers/storage]trunk.registry.rdoproject.org/tripleomaster/openstack-base:f527c9f797367c5c38e5f308573cd318@165e5d02e7eb3068e41031ace4204b3fd8c1527e5cef1755187609925e1f970a"\nlevel=debug msg=" Using transport \\"containers-storage\\" policy section \\"\\""\nlevel=debug msg=" Requirement 0: allowed"\nlevel=debug msg="Overall: allowed"\nGetting image source signatures\nlevel=debug msg="Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.docker.distribution.manifest.v1+json]"\nlevel=debug msg="... will first try using the original manifest unmodified"\nlevel=debug msg="Checking /v2/tripleomaster/openstack-base/blobs/sha256:ccf04fbd6e1943f648d1c2980e96038edc02b543c597556098ab2bcaa4fd1fa8"\nlevel=debug msg="GET https://trunk.registry.rdoproject.org/v2/"\nlevel=debug msg="Checking /v2/tripleomaster/openstack-base/blobs/sha256:77b819973c6a068f95b8f5f0523dfb9f28a3f656e744d504bbe8718be45a71f6"\nlevel=debug msg="Checking /v2/tripleomaster/openstack-base/blobs/sha256:b7b591e3443f17f9d8272b8d118b6c031ca826deb09d4b44f296ba934f1b6e57"\nlevel=debug msg="Ping https://trunk.registry.rdoproject.org/v2/ status 401"\nlevel=debug msg="GET https://trunk.registry.rdoproject.org/openshift/token?account=unused&scope=repository%3Atripleomaster%2Fopenstack-base%3Apull%2Cpush"\nlevel=debug msg="GET https://trunk.registry.rdoproject.org/openshift/token?account=unused&scope=repository%3Atripleomaster%2Fopenstack-base%3Apull%2Cpush"\nlevel=debug msg="GET https://trunk.registry.rdoproject.org/openshift/token?account=unused&scope=repository%3Atripleomaster%2Fopenstack-base%3Apull%2Cpush"\nlevel=debug msg="Increasing token expiration to: 60 seconds"\nlevel=debug msg="HEAD https://trunk.registry.rdoproject.org/v2/tripleomaster/openstack-base/blobs/sha256:b7b591e3443f17f9d8272b8d118b6c031ca826deb09d4b44f296ba934f1b6e57"\nlevel=debug msg="Increasing token expiration to: 60 seconds"\nlevel=debug msg="HEAD https://trunk.registry.rdoproject.org/v2/tripleomaster/openstack-base/blobs/sha256:ccf04fbd6e1943f648d1c2980e96038edc02b543c597556098ab2bcaa4fd1fa8"\nlevel=debug msg="Increasing token expiration to: 60 seconds"\nlevel=debug msg="HEAD https://trunk.registry.rdoproject.org/v2/tripleomaster/openstack-base/blobs/sha256:77b819973c6a068f95b8f5f0523dfb9f28a3f656e744d504bbe8718be45a71f6"\nlevel=debug msg="... not present"\nlevel=debug msg="exporting filesystem layer \\"ccf04fbd6e1943f648d1c2980e96038edc02b543c597556098ab2bcaa4fd1fa8\\" without compression for blob \\"sha256:ccf04fbd6e1943f648d1c2980e96038edc02b543c597556098ab2bcaa4fd1fa8\\""\nlevel=debug msg="... not present"\nlevel=debug msg="exporting filesystem layer \\"c48d55ebf58fab22f62cd75b2dfac6b96ef981d02a9929f3c8cdfbc1408ee53b\\" without compression for blob \\"sha256:b7b591e3443f17f9d8272b8d118b6c031ca826deb09d4b44f296ba934f1b6e57\\""\nlevel=debug msg="... not present"\nlevel=debug msg="exporting filesystem layer \\"591c59d6d8bddacb883b72bdd4779dc4d1df2f5234ab1dd4a671900d5db4ea95\\" without compression for blob \\"sha256:77b819973c6a068f95b8f5f0523dfb9f28a3f656e744d504bbe8718be45a71f6\\""\nCopying blob sha256:b7b591e3443f17f9d8272b8d118b6c031ca826deb09d4b44f296ba934f1b6e57\nlevel=debug msg="No compression detected"\nlevel=debug msg="Compressing blob on the fly"\nlevel=debug msg="Uploading /v2/tripleomaster/openstack-base/blobs/uploads/"\nlevel=debug msg="POST https://trunk.registry.rdoproject.org/v2/tripleomaster/openstack-base/blobs/uploads/"\nlevel=debug msg="Error initiating layer upload, response http.Response{Status:\\"500 Internal Server Error\\", StatusCode:500, Proto:\\"HTTP/1.1\\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\\"Content-Length\\":[]string{\\"256\\"}, \\"Content-Type\\":[]string{\\"application/json; charset=utf-8\\"}, \\"Date\\":[]string{\\"Mon, 27 Dec 2021 00:32:15 GMT\\"}, \\"Docker-Distribution-Api-Version\\":[]string{\\"registry/2.0\\"}, \\"Set-Cookie\\":[]string{\\"172555eec50a0d95563a405b15a8a45f=4c2d2b008a0709adeb6721d3c4a86c52; path=/; HttpOnly; Secure; SameSite=None\\"}, \\"X-Registry-Supports-Signatures\\":[]string{\\"1\\"}}, Body:(*http.bodyEOFSignal)(0xc0003001c0), ContentLength:256, TransferEncoding:[]string(nil), Close:true, Uncompressed:false, Trailer:http.Header(nil), Request:(*http.Request)(0xc000112100), TLS:(*tls.ConnectionState)(0xc0001f40b0)}"\nCopying blob sha256:ccf04fbd6e1943f648d1c2980e96038edc02b543c597556098ab2bcaa4fd1fa8\nlevel=debug msg="No compression detected"\nlevel=debug msg="Compressing blob on the fly"\nlevel=debug msg="Uploading /v2/tripleomaster/openstack-base/blobs/uploads/"\nlevel=debug msg="POST https://trunk.registry.rdoproject.org/v2/tripleomaster/openstack-base/blobs/uploads/"\nlevel=debug msg="Error initiating layer upload, response http.Response{Status:\\"500 Internal Server Error\\", StatusCode:500, Proto:\\"HTTP/1.1\\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\\"Content-Length\\":[]string{\\"256\\"}, \\"Content-Type\\":[]string{\\"application/json; charset=utf-8\\"}, \\"Date\\":[]string{\\"Mon, 27 Dec 2021 00:32:15 GMT\\"}, \\"Docker-Distribution-Api-Version\\":[]string{\\"registry/2.0\\"}, \\"Set-Cookie\\":[]string{\\"172555eec50a0d95563a405b15a8a45f=4c2d2b008a0709adeb6721d3c4a86c52; path=/; HttpOnly; Secure; SameSite=None\\"}, \\"X-Registry-Supports-Signatures\\":[]string{\\"1\\"}}, Body:(*http.bodyEOFSignal)(0xc00025a0c0), ContentLength:256, TransferEncoding:[]string(nil), Close:true, Uncompressed:false, Trailer:http.Header(nil), Request:(*http.Request)(0xc000112100), TLS:(*tls.ConnectionState)(0xc0000f26e0)}"\nCopying blob sha256:77b819973c6a068f95b8f5f0523dfb9f28a3f656e744d504bbe8718be45a71f6\nlevel=debug msg="No compression detected"\nlevel=debug msg="Compressing blob on the fly"\nlevel=debug msg="Uploading /v2/tripleomaster/openstack-base/blobs/uploads/"\nlevel=debug msg="POST https://trunk.registry.rdoproject.org/v2/tripleomaster/openstack-base/blobs/uploads/"\nlevel=debug msg="Error initiating layer upload, response http.Response{Status:\\"500 Internal Server Error\\", StatusCode:500, Proto:\\"HTTP/1.1\\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\\"Content-Length\\":[]string{\\"256\\"}, \\"Content-Type\\":[]string{\\"application/json; charset=utf-8\\"}, \\"Date\\":[]string{\\"Mon, 27 Dec 2021 00:32:15 GMT\\"}, \\"Docker-Distribution-Api-Version\\":[]string{\\"registry/2.0\\"}, \\"Set-Cookie\\":[]string{\\"172555eec50a0d95563a405b15a8a45f=4c2d2b008a0709adeb6721d3c4a86c52; path=/; HttpOnly; Secure; SameSite=None\\"}, \\"X-Registry-Supports-Signatures\\":[]string{\\"1\\"}}, Body:(*http.bodyEOFSignal)(0xc000300700), ContentLength:256, TransferEncoding:[]string(nil), Close:true, Uncompressed:false, Trailer:http.Header(nil), Request:(*http.Request)(0xc000112200), TLS:(*tls.ConnectionState)(0xc000272000)}"\nlevel=debug msg="parsed reference into \\"[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]trunk.registry.rdoproject.org/tripleomaster/openstack-base:f527c9f797367c5c38e5f308573cd318\\""\nerror pushing image "trunk.registry.rdoproject.org/tripleomaster/openstack-base:f527c9f797367c5c38e5f308573cd318" to "docker://trunk.registry.rdoproject.org/tripleomaster/openstack-base:f527c9f797367c5c38e5f308573cd318": error copying layers and metadata from "containers-storage:[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]trunk.registry.rdoproject.org/tripleomaster/openstack-base:f527c9f797367c5c38e5f308573cd318" to "docker://trunk.registry.rdoproject.org/tripleomaster/openstack-base:f527c9f797367c5c38e5f308573cd318": Error writing blob: Error initiating layer upload to /v2/tripleomaster/openstack-base/blobs/uploads/ in trunk.registry.rdoproject.org: received unexpected HTTP status: 500 Internal Server Error\nlevel=debug msg="shutting down the store"\n' clean_up /home/zuul/workspace/venv_build/lib64/python3.6/site-packages/osc_lib/shell.py:496
2021-12-27 00:32:15 | 2021-12-27 00:32:15.723 42351 INFO osc_lib.shell [-] END return value: 1

# links:

[1]: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-8-push/c2b6152/job-output.txt

[2]: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-8-push/c2b6152/logs/build.log

[3]: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-build-containers-ubi-8-push&job_name=periodic-tripleo-ci-build-containers-ubi-8-push-wallaby&job_name=periodic-tripleo-ci-build-containers-ubi-8-push-victoria&job_name=periodic-tripleo-ci-build-containers-ubi-8-push-ussuri&job_name=periodic-tripleo-ci-build-containers-ubi-8-push-train

[Rabi Mishra (rabi)]

Sounds like issue with rdo registry.

[chandan kumar (chkumar246)]

Based on my local testing, it seems to be an issue with RDO registry.
```
[chandankumar@fedora testproject]$ podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
quay.io/centos/centos stream9 8f3ace74d41b 3 weeks ago 154 MB
docker.io/library/registry 2 b8604a3fe854 6 weeks ago 26.8 MB
registry.access.redhat.com/ubi8 latest cc0656847854 7 weeks ago 235 MB
docker-registry.upshift.redhat.com/ubi9-beta/ubi latest 28b0a4b69d9b 2 months ago 229 MB
registry.access.redhat.com/ubi9-beta/ubi latest 28b0a4b69d9b 2 months ago 229 MB
[chandankumar@fedora testproject]$ podman push 8f3ace74d41b docker://trunk.registry.rdoproject.org/ceph/centos:stream9
Getting image source signatures
Copying blob 4448dd7f82eb [--------------------------------------] 8.0b / 146.6MiB
Error: writing blob: initiating layer upload to /v2/ceph/centos/blobs/uploads/ in trunk.registry.rdoproject.org: received unexpected HTTP status: 500 Internal Server Error
[chandankumar@fedora testproject]$

```

[Bhagyashri Shewale (bhagyashri-shewale)]

guilhermesp_> hi team. we are looking at it now
<guilhermesp_> ykarel: dpawlik can you check the systems now?
<ykarel> guilhermesp_, looking
<ykarel> i see bridge.soft is up
<ykarel> others seems still down
<ykarel> like https://images.rdoproject.org/
<guilhermesp_> ok do you have a list of all servers in the same situation now?
<ykarel> i can try to find but may be miss some
<guilhermesp_> ok im doing some more checks on my side meanwhile
<ykarel> guilhermesp_, aren't all our servers on same tenant?
<guilhermesp_> yeah, but initially we noticed a kernel bug on one of the controllers which was preventing the active l3 agent to be released or/and to be forwarding packages
<ykarel> ahhk
<ykarel> so now i can see those infra servers in infra-rdo tenant
<ykarel> and seems they are up
<ykarel> but not reachable
<guilhermesp_> ack
<ykarel> ex trunk-centos8.rdoproject.org
<ykarel> trunk-centos7.rdoproject.org
<ykarel> logserver.rdoproject.org
<ykarel> mirror.regionone.vexxhost.rdoproject.org
<ykarel> so almost all of those
<guilhermesp_> can you check those on infra-rdo now ykarel ?
<ykarel> seems you rebooted bridge.softwarefactory-project.io, right?
<ykarel> ^ is in infra-sf tenant
<guilhermesp_> hum we havent rebooted any server by now, we just fallback some l3 agents
<ykarel> ahhk, may be someone else tried as i see time Dec. 28, 2021, 8:06 a.m.
<guilhermesp_> yep i remember i saw something in the ticket description abouit that
<guilhermesp_> i believe infra-rdo servers should be back
<ykarel> guilhermesp_, me checks
<ykarel> yes seems they are getting up, i see gerrit/zuul are up atleast
<ykarel> https://review.rdoproject.org/zuul/status https://review.rdoproject.org/r
<ykarel> https://images.rdoproject.org/, https://logserver.rdoproject.org/ too
<guilhermesp_> yeah we have recovered most of the routers by now
<guilhermesp_> and we are remediating one of the controllers now
<ykarel> but can't ssh to those servers for some reason, need to check why
<guilhermesp_> huuum weird are they responding to 22?
<ykarel> yes responding
<ykarel> seems some local issue related to socket
<guilhermesp_> what do you see when trying to login to those servers?
<guilhermesp_> ah ok
<guilhermesp_> yeah i think we have fixed the access by now
<guilhermesp_> and we took off this bad controller out of the pool to fix it
<ykarel> guilhermesp_, from bridge node it said
<ykarel> unix_listener: cannot bind to path /<email address hidden>:22.fZjvgwJHmYEKYvVd: No such file or directory
<ykarel> so i created /run/user/1000 and added owner, and now i can ssh
<guilhermesp_> ok good
<ykarel> will note it down and will check with team once they are back, may be it's some known issue
<ykarel> for the router specific issue you can update the ticket as they would be needing all these details
<guilhermesp_> sure i will now

[chandan kumar (chkumar246)]

Based on the current testing, with the effort of ykarel and Daniel, everything is back to normal.
https://review.rdoproject.org/r/c/testproject/+/37505/1#message-10b0004bc152fa8a66d6b90887c5e3b529642b5c

```
periodic-tripleo-ci-build-containers-ubi-8-push https://review.rdoproject.org/zuul/build/58ebfc1736834d7d9fdaf6a095639c8a : SUCCESS in 1h 06m 22s
periodic-tripleo-ci-build-containers-ubi-8-push-wallaby https://review.rdoproject.org/zuul/build/804576de433f42faa4c0a3a2666199c4 : SUCCESS in 56m 51s
periodic-tripleo-ci-build-containers-ubi-8-push-victoria https://review.rdoproject.org/zuul/build/60bae27103da4157945a83418160aec6 : SUCCESS in 59m 08s
periodic-tripleo-ci-build-containers-ubi-8-push-ussuri https://review.rdoproject.org/zuul/build/e29246f5aa554c13920c73c65c6d0c2b : SUCCESS in 55m 43s
periodic-tripleo-ci-build-containers-ubi-8-push-train https://review.rdoproject.org/zuul/build/98fc56dad4c24c5884286913c9cc7fb8 : SUCCESS in 46m 39s
periodic-tripleo-ci-build-containers-centos-9-push-master https://review.rdoproject.org/zuul/build/2bb69645f4ed4156bdc788eac77849d2 : SUCCESS in 40m 37s
```

I think we can close this bug now.