Container build on failures reports Exception information: 'int' object is not callable

[yatin]

On Container build failures, which container failed to build is not reported instead it fails with below exception:-
Exception occured while running the command
Traceback (most recent call last):
  File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/tripleoclient/command.py", line 34, in run
    super(Command, self).run(parsed_args)
  File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run
    return super(Command, self).run(parsed_args)
  File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/cliff/command.py", line 187, in run
    return_code = self.take_action(parsed_args) or 0
  File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/tripleoclient/v2/tripleo_container_image.py", line 649, in take_action
    bb.build_all()
  File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/tripleo_common/image/builder/buildah.py", line 214, in build_all
    self._multi_build(containers=containers)
  File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/tripleo_common/image/builder/buildah.py", line 328, in _multi_build
    exceptions='\n'.join(exceptions)
RuntimeError:
The following errors were detected during container build(s):

Exception information: 'int' object is not callable

The following errors were detected during container build(s):

Exception information: 'int' object is not callable
Traceback (most recent call last):
  File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/tripleoclient/command.py", line 34, in run
    super(Command, self).run(parsed_args)
  File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run
    return super(Command, self).run(parsed_args)
  File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/cliff/command.py", line 187, in run
    return_code = self.take_action(parsed_args) or 0
  File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/tripleoclient/v2/tripleo_container_image.py", line 649, in take_action
    bb.build_all()
  File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/tripleo_common/image/builder/buildah.py", line 214, in build_all
    self._multi_build(containers=containers)
  File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/tripleo_common/image/builder/buildah.py", line 328, in _multi_build
    exceptions='\n'.join(exceptions)
RuntimeError:
The following errors were detected during container build(s):

Exception information: 'int' object is not callable
clean_up Build:
The following errors were detected during container build(s):

Exception information: 'int' object is not callable
END return value: 1

Example log:- https://a6170f8838bfb2ea8967-dcc4c568f423fe000e95109bdebe1978.ssl.cf1.rackcdn.com/761702/1/check/tripleo-build-containers-ubi-8/af672e4/logs/containers-build-errors.log
Actually rsyslog failed:- https://a6170f8838bfb2ea8967-dcc4c568f423fe000e95109bdebe1978.ssl.cf1.rackcdn.com/761702/1/check/tripleo-build-containers-ubi-8/af672e4/logs/buildah-builds/container-builds/0f71a32e-8552-459f-b407-4673f1894025/base/rsyslog/rsyslog-build.log

https://logserver.rdoproject.org/ci.centos.org/tripleo-upstream-containers-build-master-ppc64le/2845/logs/logs/build.log
Actually tempest failed:- https://logserver.rdoproject.org/ci.centos.org/tripleo-upstream-containers-build-master-ppc64le/2845/logs/logs/container-builds/container-builds/e6240e68-f1c7-49e5-90ac-84fe956313af/base/os/tempest/tempest-build.log

[Alex Schultz (alex-schultz)]

The error is coming out of tenacity.

2020-11-12 17:25:13.950 344463 ERROR tripleo_common.image.builder.buildah.BuildahBuilder Traceback (most recent call last):
2020-11-12 17:25:13.950 344463 ERROR tripleo_common.image.builder.buildah.BuildahBuilder File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/tripleo_common/image/builder/buildah.py", line 136, in _generate_container
2020-11-12 17:25:13.950 344463 ERROR tripleo_common.image.builder.buildah.BuildahBuilder self.build(container_name, self._find_container_dir(container_name))
2020-11-12 17:25:13.950 344463 ERROR tripleo_common.image.builder.buildah.BuildahBuilder File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/tenacity/__init__.py", line 329, in wrapped_f
2020-11-12 17:25:13.950 344463 ERROR tripleo_common.image.builder.buildah.BuildahBuilder return self.call(f, *args, **kw)
2020-11-12 17:25:13.950 344463 ERROR tripleo_common.image.builder.buildah.BuildahBuilder File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/tenacity/__init__.py", line 409, in call
2020-11-12 17:25:13.950 344463 ERROR tripleo_common.image.builder.buildah.BuildahBuilder do = self.iter(retry_state=retry_state)
2020-11-12 17:25:13.950 344463 ERROR tripleo_common.image.builder.buildah.BuildahBuilder File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/tenacity/__init__.py", line 372, in iter
2020-11-12 17:25:13.950 344463 ERROR tripleo_common.image.builder.buildah.BuildahBuilder sleep = self.wait(retry_state=retry_state)
2020-11-12 17:25:13.950 344463 ERROR tripleo_common.image.builder.buildah.BuildahBuilder TypeError: 'int' object is not callable

Likely caused by https://review.opendev.org/#/c/761609/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/762558

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/762571

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/762558
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=52112d428defc2d2d03ba7a0b261cc4c51f50765
Submitter: Zuul
Branch: master

commit 52112d428defc2d2d03ba7a0b261cc4c51f50765
Author: Alex Schultz <email address hidden>
Date: Thu Nov 12 10:53:22 2020 -0700

    Fix wait condition for retry

    Tenancity expects you use one it's waiting methods and not a fixed
    number. This currently doesn't retry and stops all builds from
    continuing to run.

    Change-Id: Ia2b61cb698677968ef990d5577918be3aa4e2450
    Closes-Bug: #1904025

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/762658

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/762659

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/762660

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/762571
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=dbfa2399b47aa3b9fef84709ff786b9bf69bf2d3
Submitter: Zuul
Branch: master

commit dbfa2399b47aa3b9fef84709ff786b9bf69bf2d3
Author: Alex Schultz <email address hidden>
Date: Thu Nov 12 12:09:19 2020 -0700

    Add better exception logging for builds

    When the exception is bubbled up through the multiprocessing bits, it
    loses some information so it's hard to troubleshoot. Let's log the
    exception in the multiprocessed function call and continue to raise the
    exception when things fail.

    Change-Id: Idbf56d97069e238bc6da61c7b3432fe37ecaf1a0
    Related-Bug: #1904025

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/762890

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/762891

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/762892

[wes hayutin (weshayutin)]

this is blocking c8 train atm https://review.rdoproject.org/zuul/build/eea5f05e9af349109dadf911225315cc

[wes hayutin (weshayutin)]

w/ this patch on train..

still get

2020-12-10 17:25:34 | Exception information: 'BuildahBuilder' object has no attribute 'log'
2020-12-10 17:25:34 | Traceback (most recent call last):
2020-12-10 17:25:34 | File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/osc_lib/shell.py", line 136, in run
2020-12-10 17:25:34 | ret_val = super(OpenStackShell, self).run(argv)
2020-12-10 17:25:34 | File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/cliff/app.py", line 281, in run
2020-12-10 17:25:34 | result = self.run_subcommand(remainder)
2020-12-10 17:25:34 | File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/osc_lib/shell.py", line 176, in run_subcommand
2020-12-10 17:25:34 | ret_value = super(OpenStackShell, self).run_subcommand(argv)
2020-12-10 17:25:34 | File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/cliff/app.py", line 401, in run_subcommand
2020-12-10 17:25:34 | result = cmd.run(parsed_args)
2020-12-10 17:25:34 | File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run
2020-12-10 17:25:34 | super(Command, self).run(parsed_args)
2020-12-10 17:25:34 | File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
2020-12-10 17:25:34 | return super(Command, self).run(parsed_args)
2020-12-10 17:25:34 | File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/cliff/command.py", line 185, in run
2020-12-10 17:25:34 | return_code = self.take_action(parsed_args) or 0
2020-12-10 17:25:34 | File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/tripleoclient/v1/tripleo_container_image.py", line 656, in take_action
2020-12-10 17:25:34 | bb.build_all()
2020-12-10 17:25:34 | File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/tripleo_common/image/builder/buildah.py", line 234, in build_all
2020-12-10 17:25:34 | exceptions='\n'.join(exceptions)
2020-12-10 17:25:34 | RuntimeError:

testing here: https://review.rdoproject.org/r/#/c/31336/

[Marios Andreou (marios-b)]

still waiting for the train cherrypick https://review.opendev.org/c/openstack/tripleo-common/+/762660/ fighting zuul in the gate

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 13.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 14.0.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 12.4.3 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 11.5.0 release.