debian: build-tools: python exception

[Davlet Panech]

Brief Description
-----------------
The following build failed on CENGN due to a python exception (?) in build-pkgs:

http://mirror.starlingx.cengn.ca/mirror/starlingx/master/debian/monolithic/20220825T060000Z/logs/build-packages.log.txt

However a more recent build succeeded.

Suggest to analyze the code in the stack trace and make sure this never happens again.

Severity
--------
Provide the severity of the defect.
Major

Steps to Reproduce
------------------
Run build-pkgs

Expected Behavior
------------------
build-pkgs runs w/o python exception

Actual Behavior
----------------
build-pkgs fails with a python exception.

Reproducibility
---------------
Seen once

System Configuration
--------------------
N/A

Branch/Pull Time/Commit
-----------------------
master/2022-08-25

Last Pass
---------
master/2022-08-23

Timestamp/Logs
--------------
http://mirror.starlingx.cengn.ca/mirror/starlingx/master/debian/monolithic/20220825T060000Z/logs/build-packages.log.txt

[2022-08-25T11:12:05.176Z] 2022-08-25 11:12:04,768 - dsc_depend - DEBUG: 10 Build_able packages, try to get 1 From them
[2022-08-25T11:12:05.176Z] 2022-08-25 11:12:04,768 - dsc_depend - DEBUG: /localdisk/loadbuild/jenkins/debian-master/std/python-neutronclient/python-neutronclient_7.2.1-2.stx.2.dsc
[2022-08-25T11:12:05.176Z] 2022-08-25 11:12:04,768 - dsc_depend - DEBUG: 1 packages will be built
[2022-08-25T11:12:05.176Z] 2022-08-25 11:12:04,768 - dsc_depend - DEBUG: ['/localdisk/loadbuild/jenkins/debian-master/std/python-neutronclient/python-neutronclient_7.2.1-2.stx.2.dsc']
[2022-08-25T11:12:05.176Z] 2022-08-25 11:12:04,769 - debcontroller - INFO: Depends resolver told to build python-neutronclient
[2022-08-25T11:12:05.176Z] 2022-08-25 11:12:04,770 - debcontroller - INFO: Clean data(stamp and build output) to prepare to build python-neutronclient
[2022-08-25T11:12:05.176Z] 2022-08-25 11:12:04,774 - debcontroller - DEBUG: Successfully removed the old deb packages and log link
[2022-08-25T11:12:05.176Z] 2022-08-25 11:12:04,774 - debcontroller - DEBUG: Repository snapshot 0 is applied for /localdisk/loadbuild/jenkins/debian-master/std/python-neutronclient/python-neutronclient_7.2.1-2.stx.2.dsc
[2022-08-25T11:12:05.176Z] 2022-08-25 11:12:04,774 - debcontroller - DEBUG: Try to deploy the repository deb-local-build-0
[2022-08-25T11:12:05.426Z] 2022-08-25 11:12:04,807 - repo_manager - DEBUG: deb-local-build find, can be used
[2022-08-25T11:12:08.655Z] 2022-08-25 11:12:08,141 - debcontroller - INFO: Successfully deployed the repository deb-local-build-0
[2022-08-25T11:12:08.655Z] 2022-08-25 11:12:08,141 - debcontroller - INFO: To Require to add build task for python-neutronclient with snapshot 0
[2022-08-25T11:12:08.655Z] 2022-08-25 11:12:08,145 - debcontroller - DEBUG: Requires the number of jobs 12 for python-neutronclient
[2022-08-25T11:12:08.655Z] Traceback (most recent call last):
[2022-08-25T11:12:08.655Z] File "/localdisk/designer/jenkins/debian-master/cgcs-root/build-tools/stx/build-pkgs", line 1393, in <module>
[2022-08-25T11:12:08.655Z] build_controller.build_all(layers=layers, build_types=build_types, packages=packages)
[2022-08-25T11:12:08.655Z] File "/localdisk/designer/jenkins/debian-master/cgcs-root/build-tools/stx/build-pkgs", line 1037, in build_all
[2022-08-25T11:12:08.655Z] self.build_layers(layers=layers, build_types=build_types, packages=packages)
[2022-08-25T11:12:08.655Z] File "/localdisk/designer/jenkins/debian-master/cgcs-root/build-tools/stx/build-pkgs", line 1164, in build_layers
[2022-08-25T11:12:08.655Z] self.build_layer(layer=layer, build_types=build_types, packages=packages)
[2022-08-25T11:12:08.655Z] File "/localdisk/designer/jenkins/debian-master/cgcs-root/build-tools/stx/build-pkgs", line 1139, in build_layer
[2022-08-25T11:12:08.655Z] self.build_layer_and_build_types(layer=layer, build_types=build_types, packages=packages)
[2022-08-25T11:12:08.655Z] File "/localdisk/designer/jenkins/debian-master/cgcs-root/build-tools/stx/build-pkgs", line 1124, in build_layer_and_build_types
[2022-08-25T11:12:08.655Z] self.build_layer_and_build_type(layer=layer, build_type=build_type, packages=packages)
[2022-08-25T11:12:08.655Z] File "/localdisk/designer/jenkins/debian-master/cgcs-root/build-tools/stx/build-pkgs", line 1089, in build_layer_and_build_type
[2022-08-25T11:12:08.655Z] self.build_packages(layer_pkg_dirs, pkg_dirs, layer, build_type=build_type)
[2022-08-25T11:12:08.655Z] File "/localdisk/designer/jenkins/debian-master/cgcs-root/build-tools/stx/build-pkgs", line 1219, in build_packages
[2022-08-25T11:12:08.655Z] self.run_build_loop(layer_pkgdir_dscs, target_pkgdir_dscs, layer, build_type=build_type)
[2022-08-25T11:12:08.655Z] File "/localdisk/designer/jenkins/debian-master/cgcs-root/build-tools/stx/build-pkgs", line 910, in run_build_loop
[2022-08-25T11:12:08.655Z] if 'ServerError' in chroot:
[2022-08-25T11:12:08.655Z] TypeError: argument of type 'NoneType' is not iterable
[2022-08-25T11:12:08.655Z] command terminated with exit code 1
[2022-08-25T11:12:08.655Z]
[2022-08-25T11:12:08.655Z]
[2022-08-25T11:12:08.655Z] ###
[2022-08-25T11:12:08.655Z] ### Failed to build packages after 3 iterations
[2022-08-25T11:12:08.655Z] ###
[2022-08-25T11:12:08.655Z]

Test Activity
-------------
N/A

Workaround
----------
N/A

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/root/+/855124

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/root/+/855124
Committed: https://opendev.org/starlingx/root/commit/50238fcead98df13f772d2df57641a9324b5656f
Submitter: "Zuul (22348)"
Branch: master

commit 50238fcead98df13f772d2df57641a9324b5656f
Author: Haiqing Bai <email address hidden>
Date: Tue Aug 30 15:25:29 2022 +0800

    build-pkgs: Fixed the issue that returned chroot is None

    If pkgbuilder fails to execute the builder's commands
    and fails to assign idle chroot for package build,
    not only the 'status' in the returned value should be
    set to 'fail', but also the 'chroot' should be used to
    set to simple words to show the error type.
    And the code is improved to use the 'chroot' safely.

    Test Plan:
    Only run 'build-pkgs -a' multiple times for this issue
    hard to reproduce.

    Closes-bug: 1987906
    Signed-off-by: Haiqing Bai <email address hidden>
    Change-Id: Ia53860a983119ddedac870c39c840e2e1377b46d