stx-8.0: build: docker image builds silently failing on Jenkins

[Davlet Panech]

Brief Description
-----------------
The 8.0 builds' docker images step has been failing since May, and Jenkins has been falsely reporting these builds as successful. The affected builds are under rc/8.0 on the download site: https://mirror.starlingx.windriver.com/mirror/starlingx/rc/8.0/

I can confirm the official 8.0 release was OK (all docker image builds succeeded). Based on a build from 2023-02-11. Ie this URL is good: https://mirror.starlingx.windriver.com/mirror/starlingx/release/8.0.0/

Reason: Early-ish in the Debian migration process many docker images were failing. Jenkinsfile was tweaked temporarily to ignore docker image errors, basically. I guess we created that branch before I had a chance to undo this behavior.

Effects:
1) timestamped 8.0 builds appear green in Jenkins, but do NOT include the failed docker images
2) The latest_build & latest_docker_image build links under rc/8.0 point to old builds: https://mirror.starlingx.windriver.com/mirror/starlingx/rc/8.0/debian/monolithic/

Severity
--------
Major

Steps to Reproduce
------------------
Try to do a Jenkins build in stx/8.0 branch

Expected Behavior
------------------
Build should fail because some docker images fail.

Actual Behavior
----------------
Build "succeeds", but you can see error messages in the docker image build log.

Reproducibility
---------------
Reproducible.

System Configuration
--------------------
N/A

Branch/Pull Time/Commit
-----------------------
master/2023-10-05

Last Pass
---------
This behaviour has been in the 8.0 branch from the beginning.

Timestamp/Logs
--------------
N/A

Test Activity
-------------
N/A

Workaround
----------
N/A

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: r/stx.8.0
Review: https://review.opendev.org/c/starlingx/jenkins-pipelines/+/897867

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/jenkins-pipelines/+/897867
Committed: https://opendev.org/starlingx/jenkins-pipelines/commit/a579327e43ccb83caf7d9f60af1e01f45e40c0c8
Submitter: "Zuul (22348)"
Branch: r/stx.8.0

commit a579327e43ccb83caf7d9f60af1e01f45e40c0c8
Author: Davlet Panech <email address hidden>
Date: Fri Feb 3 10:25:41 2023 -0500

    Better error handling for docker image sub-jobs

    Currently Jenkinsfile ignores failures in sub-jobs related to docker
    images (base image, wheels, etc). If they fail, the overall build
    remains successful. This allows docker-related build problems remain
    unnoticed.

    Make this behavior conditional on a job parameter.

    TESTS
    ==============================================
    * A build where there are no docker-related errors (overall: SUCCESS)
    * A build where wheels fail and SUPPRESS_DOCKER_IMAGE_BUILD_ERRORS is
      true (overall: SUCCESS)
    * A build where wheels fail and SUPPRESS_DOCKER_IMAGE_BUILD_ERRORS is
      false (overall: FAIL)

    Story: 2009895
    Task: 47280
    Closes-Bug: 2038569

    Signed-off-by: Davlet Panech <email address hidden>
    Change-Id: I16ac4f63033ecbc78c647c4da105d550fe2facc4
    (cherry picked from commit 7481a3b257596c34b00af830e263f765e297733f)

[Davlet Panech (dpanech)]

Latest build succeeded, including docker images:

https://mirror.starlingx.windriver.com/mirror/starlingx/rc/8.0/debian/monolithic/20231014T060059Z/logs/build-docker-images-stable.log.txt