Tripleo build image command fails to build some images

[Arx Cruz]

As can be seen in https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable1-cs8

The jobs periodic-tripleo-ci-build-containers-ubi-8-push-wallaby and periodic-tripleo-ci-build-containers-ubi-8-quay-push-wallaby are failing because of missing containers such as nova-api, nova-scheduler among others.

It is hard right now to find the root cause because there is no logs for these builds, only at the end of the build containers log, you can notice this:

2022-09-05 14:09:31 | 2022-09-05 14:09:31.610 39724 ERROR tripleo_common.image.builder.buildah.BuildahBuilder [-] [Errno 2] No such file or directory: '/home/zuul/container-builds/9e4b8ca8-1450-485b-8eb7-dfa2b42b0c35/base/os/swift-base/swift-proxy-server/swift-proxy-server-build.log': FileNotFoundError: [Errno 2] No such file or directory: '/home/zuul/container-builds/9e4b8ca8-1450-485b-8eb7-dfa2b42b0c35/base/os/swift-base/swift-proxy-server/swift-proxy-server-build.log'
2022-09-05 14:09:31 | 2022-09-05 14:09:31.610 39724 ERROR tripleo_common.image.builder.buildah.BuildahBuilder Traceback (most recent call last):
2022-09-05 14:09:31 | 2022-09-05 14:09:31.610 39724 ERROR tripleo_common.image.builder.buildah.BuildahBuilder File "/home/zuul/workspace/venv_build/lib64/python3.6/site-packages/tripleo_common/image/builder/buildah.py", line 143, in _generate_container
2022-09-05 14:09:31 | 2022-09-05 14:09:31.610 39724 ERROR tripleo_common.image.builder.buildah.BuildahBuilder self._find_container_dir(container_name))
2022-09-05 14:09:31 | 2022-09-05 14:09:31.610 39724 ERROR tripleo_common.image.builder.buildah.BuildahBuilder File "/home/zuul/workspace/venv_build/lib64/python3.6/site-packages/tenacity/__init__.py", line 333, in wrapped_f
2022-09-05 14:09:31 | 2022-09-05 14:09:31.610 39724 ERROR tripleo_common.image.builder.buildah.BuildahBuilder return self(f, *args, **kw)
2022-09-05 14:09:31 | 2022-09-05 14:09:31.610 39724 ERROR tripleo_common.image.builder.buildah.BuildahBuilder File "/home/zuul/workspace/venv_build/lib64/python3.6/site-packages/tenacity/__init__.py", line 423, in __call__
2022-09-05 14:09:31 | 2022-09-05 14:09:31.610 39724 ERROR tripleo_common.image.builder.buildah.BuildahBuilder do = self.iter(retry_state=retry_state)
2022-09-05 14:09:31 | 2022-09-05 14:09:31.610 39724 ERROR tripleo_common.image.builder.buildah.BuildahBuilder File "/home/zuul/workspace/venv_build/lib64/python3.6/site-packages/tenacity/__init__.py", line 372, in iter
2022-09-05 14:09:31 | 2022-09-05 14:09:31.610 39724 ERROR tripleo_common.image.builder.buildah.BuildahBuilder raise retry_exc.reraise()
2022-09-05 14:09:31 | 2022-09-05 14:09:31.610 39724 ERROR tripleo_common.image.builder.buildah.BuildahBuilder File "/home/zuul/workspace/venv_build/lib64/python3.6/site-packages/tenacity/__init__.py", line 189, in reraise
2022-09-05 14:09:31 | 2022-09-05 14:09:31.610 39724 ERROR tripleo_common.image.builder.buildah.BuildahBuilder raise self.last_attempt.result()
2022-09-05 14:09:31 | 2022-09-05 14:09:31.610 39724 ERROR tripleo_common.image.builder.buildah.BuildahBuilder File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result
2022-09-05 14:09:31 | 2022-09-05 14:09:31.610 39724 ERROR tripleo_common.image.builder.buildah.BuildahBuilder return self.__get_result()
2022-09-05 14:09:31 | 2022-09-05 14:09:31.610 39724 ERROR tripleo_common.image.builder.buildah.BuildahBuilder File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2022-09-05 14:09:31 | 2022-09-05 14:09:31.610 39724 ERROR tripleo_common.image.builder.buildah.BuildahBuilder raise self._exception
2022-09-05 14:09:31 | 2022-09-05 14:09:31.610 39724 ERROR tripleo_common.image.builder.buildah.BuildahBuilder File "/home/zuul/workspace/venv_build/lib64/python3.6/site-packages/tenacity/__init__.py", line 426, in __call__
2022-09-05 14:09:31 | 2022-09-05 14:09:31.610 39724 ERROR tripleo_common.image.builder.buildah.BuildahBuilder result = fn(*args, **kwargs)
2022-09-05 14:09:31 | 2022-09-05 14:09:31.610 39724 ERROR tripleo_common.image.builder.buildah.BuildahBuilder File "/home/zuul/workspace/venv_build/lib64/python3.6/site-packages/tripleo_common/image/builder/buildah.py", line 175, in build
2022-09-05 14:09:31 | 2022-09-05 14:09:31.610 39724 ERROR tripleo_common.image.builder.buildah.BuildahBuilder pathlib.Path(logfile).touch()
2022-09-05 14:09:31 | 2022-09-05 14:09:31.610 39724 ERROR tripleo_common.image.builder.buildah.BuildahBuilder File "/usr/lib64/python3.6/pathlib.py", line 1238, in touch
2022-09-05 14:09:31 | 2022-09-05 14:09:31.610 39724 ERROR tripleo_common.image.builder.buildah.BuildahBuilder fd = self._raw_open(flags, mode)
2022-09-05 14:09:31 | 2022-09-05 14:09:31.610 39724 ERROR tripleo_common.image.builder.buildah.BuildahBuilder File "/usr/lib64/python3.6/pathlib.py", line 1046, in _raw_open
2022-09-05 14:09:31 | 2022-09-05 14:09:31.610 39724 ERROR tripleo_common.image.builder.buildah.BuildahBuilder return self._accessor.open(self, flags, mode)
2022-09-05 14:09:31 | 2022-09-05 14:09:31.610 39724 ERROR tripleo_common.image.builder.buildah.BuildahBuilder File "/usr/lib64/python3.6/pathlib.py", line 387, in wrapped
2022-09-05 14:09:31 | 2022-09-05 14:09:31.610 39724 ERROR tripleo_common.image.builder.buildah.BuildahBuilder return strfunc(str(pathobj), *args)
2022-09-05 14:09:31 | 2022-09-05 14:09:31.610 39724 ERROR tripleo_common.image.builder.buildah.BuildahBuilder FileNotFoundError: [Errno 2] No such file or directory: '/home/zuul/container-builds/9e4b8ca8-1450-485b-8eb7-dfa2b42b0c35/base/os/swift-base/swift-proxy-server

However, since we can't find the logs for these builds, it's hard to identify the reason it is failing.

[Arx Cruz (arxcruz)]

I tried to run the job in a testproject and hold the node to debug more, but the job passes, however, on the pipeline it is always failing with timeout.

[chandan kumar (chkumar246)]

Re-running the job on IBM cloud https://review.rdoproject.org/r/c/testproject/+/44779 and see if we can reproduce it or not.

[chandan kumar (chkumar246)]

ON ibm cloud, https://review.rdoproject.org/r/c/testproject/+/44779 periodic-tripleo-ci-build-containers-ubi-8-push-wallaby SUCCESS 53m 51s passed.

It might happened due to IBM cloud stress testing testproject https://review.rdoproject.org/r/c/testproject/+/44388

for current situation, we will be moving the content provider to vexxhost cloud. Will fallback to ibm cloud once the situation improves.

[chandan kumar (chkumar246)]

https://review.rdoproject.org/r/c/rdo-jobs/+/44788 moved the job to vexxhost. Please revert this patch and it will move the job to ibm cloud.

[Ronelle Landy (rlandy)]

https://review.rdoproject.org/r/c/rdo-jobs/+/44757 - revert added