CENGN has been unable to complete a build since March 3.
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
StarlingX |
Fix Released
|
Critical
|
Dostoievski Albino Batista |
Bug Description
Brief Description
-----------------
CENGN has been unable to complete a build since March 3.
Since then, four builds were attempted, and all builds hung within the post-build unit tests of python3.
One build was hung for nearly 48 hours.
The logs are not specific on which test is hanging.
Killing the hung unit test results in the overall build of the python3.9 package build failing, but a retry loop attempts to rebuild, and it again hangs in the unit tests.
There is nothing in the change logs that directly affect this package. The only build system changes relate to secure boot. signing and as the python package is not one that requires signing, I'm currently discounting those as a cause.
An equivalent build using an internal WindRiver build machine has so far not hit this issue. The main difference being that CENGN uses minikube, and the internal server is using kubernetes directly.
One theory was that something change upstream that affects the content within the build containers. However, both CENGN and the internal build server rebuild the build containers each time. If there was a change upstream, I would expect both builds to see it.
Most designers are likely using minikube, and so far I've seen no complaints from designers on this topic. Perhaps designers are using a build environment created on or before March 3, and haven't seen it yet.
Severity
--------
Critical
Steps to Reproduce
------------------
build-pkgs
Expected Behavior
------------------
build succeeds
Actual Behavior
----------------
build hangs
Reproducibility
---------------
Reproducible on CENGN only so far
System Configuration
-------
N/A
Branch/Pull Time/Commit
-------
March 7
Last Pass
---------
March 3
Timestamp/Logs
--------------
1:24:00 load avg: 0.91 running: test_builtin (1 hour 24 min)
1:24:30 load avg: 0.85 running: test_builtin (1 hour 24 min)
1:25:00 load avg: 0.92 running: test_builtin (1 hour 25 min)
1:25:30 load avg: 0.71 running: test_builtin (1 hour 25 min)
1:26:00 load avg: 0.61 running: test_builtin (1 hour 26 min)
1:26:30 load avg: 0.37 running: test_builtin (1 hour 26 min)
1:27:00 load avg: 0.30 running: test_builtin (1 hour 27 min)
1:27:30 load avg: 0.18 running: test_builtin (1 hour 27 min)
1:28:00 load avg: 0.37 running: test_builtin (1 hour 28 min)
1:28:30 load avg: 0.58 running: test_builtin (1 hour 28 min)
...
Test Activity
-------------
Build
Workaround
----------
disable python3.9's unit tests
Changed in starlingx: | |
importance: | Undecided → Critical |
assignee: | nobody → Dostoievski Albino Batista (dalbinob) |
tags: | added: stx.9.0 stx.build |
Reproducible from an new independent minikube build environment running on the CENGN build server.