Container build on failures reports Exception information: 'int' object is not callable

Bug #1904025 reported by yatin
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Alex Schultz

Bug Description

On Container build failures, which container failed to build is not reported instead it fails with below exception:-
Exception occured while running the command
Traceback (most recent call last):
  File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/tripleoclient/command.py", line 34, in run
    super(Command, self).run(parsed_args)
  File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run
    return super(Command, self).run(parsed_args)
  File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/cliff/command.py", line 187, in run
    return_code = self.take_action(parsed_args) or 0
  File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/tripleoclient/v2/tripleo_container_image.py", line 649, in take_action
    bb.build_all()
  File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/tripleo_common/image/builder/buildah.py", line 214, in build_all
    self._multi_build(containers=containers)
  File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/tripleo_common/image/builder/buildah.py", line 328, in _multi_build
    exceptions='\n'.join(exceptions)
RuntimeError:
The following errors were detected during container build(s):

Exception information: 'int' object is not callable

The following errors were detected during container build(s):

Exception information: 'int' object is not callable
Traceback (most recent call last):
  File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/tripleoclient/command.py", line 34, in run
    super(Command, self).run(parsed_args)
  File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run
    return super(Command, self).run(parsed_args)
  File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/cliff/command.py", line 187, in run
    return_code = self.take_action(parsed_args) or 0
  File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/tripleoclient/v2/tripleo_container_image.py", line 649, in take_action
    bb.build_all()
  File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/tripleo_common/image/builder/buildah.py", line 214, in build_all
    self._multi_build(containers=containers)
  File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/tripleo_common/image/builder/buildah.py", line 328, in _multi_build
    exceptions='\n'.join(exceptions)
RuntimeError:
The following errors were detected during container build(s):

Exception information: 'int' object is not callable
clean_up Build:
The following errors were detected during container build(s):

Exception information: 'int' object is not callable
END return value: 1

Example log:- https://a6170f8838bfb2ea8967-dcc4c568f423fe000e95109bdebe1978.ssl.cf1.rackcdn.com/761702/1/check/tripleo-build-containers-ubi-8/af672e4/logs/containers-build-errors.log
Actually rsyslog failed:- https://a6170f8838bfb2ea8967-dcc4c568f423fe000e95109bdebe1978.ssl.cf1.rackcdn.com/761702/1/check/tripleo-build-containers-ubi-8/af672e4/logs/buildah-builds/container-builds/0f71a32e-8552-459f-b407-4673f1894025/base/rsyslog/rsyslog-build.log

https://logserver.rdoproject.org/ci.centos.org/tripleo-upstream-containers-build-master-ppc64le/2845/logs/logs/build.log
Actually tempest failed:- https://logserver.rdoproject.org/ci.centos.org/tripleo-upstream-containers-build-master-ppc64le/2845/logs/logs/container-builds/container-builds/e6240e68-f1c7-49e5-90ac-84fe956313af/base/os/tempest/tempest-build.log

yatin (yatinkarel)
Changed in tripleo:
status: New → Triaged
importance: Undecided → High
milestone: none → wallaby-1
Changed in tripleo:
assignee: nobody → Alex Schultz (alex-schultz)
Revision history for this message
Alex Schultz (alex-schultz) wrote :

The error is coming out of tenacity.

2020-11-12 17:25:13.950 344463 ERROR tripleo_common.image.builder.buildah.BuildahBuilder Traceback (most recent call last):
2020-11-12 17:25:13.950 344463 ERROR tripleo_common.image.builder.buildah.BuildahBuilder File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/tripleo_common/image/builder/buildah.py", line 136, in _generate_container
2020-11-12 17:25:13.950 344463 ERROR tripleo_common.image.builder.buildah.BuildahBuilder self.build(container_name, self._find_container_dir(container_name))
2020-11-12 17:25:13.950 344463 ERROR tripleo_common.image.builder.buildah.BuildahBuilder File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/tenacity/__init__.py", line 329, in wrapped_f
2020-11-12 17:25:13.950 344463 ERROR tripleo_common.image.builder.buildah.BuildahBuilder return self.call(f, *args, **kw)
2020-11-12 17:25:13.950 344463 ERROR tripleo_common.image.builder.buildah.BuildahBuilder File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/tenacity/__init__.py", line 409, in call
2020-11-12 17:25:13.950 344463 ERROR tripleo_common.image.builder.buildah.BuildahBuilder do = self.iter(retry_state=retry_state)
2020-11-12 17:25:13.950 344463 ERROR tripleo_common.image.builder.buildah.BuildahBuilder File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/tenacity/__init__.py", line 372, in iter
2020-11-12 17:25:13.950 344463 ERROR tripleo_common.image.builder.buildah.BuildahBuilder sleep = self.wait(retry_state=retry_state)
2020-11-12 17:25:13.950 344463 ERROR tripleo_common.image.builder.buildah.BuildahBuilder TypeError: 'int' object is not callable

Likely caused by https://review.opendev.org/#/c/761609/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (master)

Fix proposed to branch: master
Review: https://review.opendev.org/762558

Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-common (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/762571

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (master)

Reviewed: https://review.opendev.org/762558
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=52112d428defc2d2d03ba7a0b261cc4c51f50765
Submitter: Zuul
Branch: master

commit 52112d428defc2d2d03ba7a0b261cc4c51f50765
Author: Alex Schultz <email address hidden>
Date: Thu Nov 12 10:53:22 2020 -0700

    Fix wait condition for retry

    Tenancity expects you use one it's waiting methods and not a fixed
    number. This currently doesn't retry and stops all builds from
    continuing to run.

    Change-Id: Ia2b61cb698677968ef990d5577918be3aa4e2450
    Closes-Bug: #1904025

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/762658

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/762659

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/762660

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-common (master)

Reviewed: https://review.opendev.org/762571
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=dbfa2399b47aa3b9fef84709ff786b9bf69bf2d3
Submitter: Zuul
Branch: master

commit dbfa2399b47aa3b9fef84709ff786b9bf69bf2d3
Author: Alex Schultz <email address hidden>
Date: Thu Nov 12 12:09:19 2020 -0700

    Add better exception logging for builds

    When the exception is bubbled up through the multiprocessing bits, it
    loses some information so it's hard to troubleshoot. Let's log the
    exception in the multiprocessed function call and continue to raise the
    exception when things fail.

    Change-Id: Idbf56d97069e238bc6da61c7b3432fe37ecaf1a0
    Related-Bug: #1904025

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-common (stable/victoria)

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/762890

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-common (stable/ussuri)

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/762891

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-common (stable/train)

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/762892

wes hayutin (weshayutin)
tags: added: promotion-blocker
Revision history for this message
wes hayutin (weshayutin) wrote :
Changed in tripleo:
status: Fix Released → In Progress
Revision history for this message
wes hayutin (weshayutin) wrote :

w/ this patch on train..

still get

2020-12-10 17:25:34 | Exception information: 'BuildahBuilder' object has no attribute 'log'
2020-12-10 17:25:34 | Traceback (most recent call last):
2020-12-10 17:25:34 | File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/osc_lib/shell.py", line 136, in run
2020-12-10 17:25:34 | ret_val = super(OpenStackShell, self).run(argv)
2020-12-10 17:25:34 | File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/cliff/app.py", line 281, in run
2020-12-10 17:25:34 | result = self.run_subcommand(remainder)
2020-12-10 17:25:34 | File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/osc_lib/shell.py", line 176, in run_subcommand
2020-12-10 17:25:34 | ret_value = super(OpenStackShell, self).run_subcommand(argv)
2020-12-10 17:25:34 | File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/cliff/app.py", line 401, in run_subcommand
2020-12-10 17:25:34 | result = cmd.run(parsed_args)
2020-12-10 17:25:34 | File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run
2020-12-10 17:25:34 | super(Command, self).run(parsed_args)
2020-12-10 17:25:34 | File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
2020-12-10 17:25:34 | return super(Command, self).run(parsed_args)
2020-12-10 17:25:34 | File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/cliff/command.py", line 185, in run
2020-12-10 17:25:34 | return_code = self.take_action(parsed_args) or 0
2020-12-10 17:25:34 | File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/tripleoclient/v1/tripleo_container_image.py", line 656, in take_action
2020-12-10 17:25:34 | bb.build_all()
2020-12-10 17:25:34 | File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/tripleo_common/image/builder/buildah.py", line 234, in build_all
2020-12-10 17:25:34 | exceptions='\n'.join(exceptions)
2020-12-10 17:25:34 | RuntimeError:

testing here: https://review.rdoproject.org/r/#/c/31336/

Revision history for this message
Marios Andreou (marios-b) wrote :

still waiting for the train cherrypick https://review.opendev.org/c/openstack/tripleo-common/+/762660/ fighting zuul in the gate

wes hayutin (weshayutin)
Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-common 13.1.0

This issue was fixed in the openstack/tripleo-common 13.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-common 14.0.0

This issue was fixed in the openstack/tripleo-common 14.0.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-common 12.4.3

This issue was fixed in the openstack/tripleo-common 12.4.3 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-common 11.5.0

This issue was fixed in the openstack/tripleo-common 11.5.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.