StarlingX

build: docker image "uft" fails to compile

Bug #2034255 reported by Davlet Panech
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
High
Rafal Lal

Bug Description

Brief Description
-----------------
Docker image "uft" fails to compile in Jenkins.

Severity
--------
Critical

Steps to Reproduce
------------------
Run the Jenkins build

Expected Behavior
------------------
Build succeeds

Actual Behavior
----------------
Docker image build step fails while compiling starlingx/uft image

Reproducibility
---------------
Reproducible

System Configuration
--------------------
N/A

Branch/Pull Time/Commit
-----------------------
master/2023-09-04

Last Pass
---------
master/2023-08-30

Timestamp/Logs
--------------

Full log: https://mirror.starlingx.cengn.ca/mirror/starlingx/master/debian/monolithic/20230904T060000Z/logs/build-docker-images-stable.log.txt

[2023-09-04T12:41:06.446Z] >>> running builder pod command:
[2023-09-04T12:41:06.446Z] cd $MY_REPO/build-tools/build-docker-images && ./build-stx-images.sh --os-label=debian --attempts=3 --retry-delay=30 --stream=stable --version=20230904T060000Z --prefix=master --user=starlingx --latest --base=starlingx/stx-debian:master-stable-20230904T060000Z --no-pull-base --wheels=$MY_WORKSPACE/std/build-wheels-debian-stable/stx-debian-stable-wheels.tar --cache --push --os=debian
[2023-09-04T12:41:06.446Z] STX-k8s - INFO: Running: kubectl -n jenkins-debian-master get pods --selector 'app.kubernetes.io/instance=debian-master,app.kubernetes.io/name=stx-builder' | awk '$3 == "Running" {print $1}' | tail -n 1
[2023-09-04T12:41:06.446Z] STX-Shell - DEBUG: Running command: kubectl -n jenkins-debian-master exec --stdin debian-master-stx-builder-7d6c8fb584-ngmng -- bash -l -c 'runuser -u $MYUNAME -- bash --norc -c '"'"'source /home/$MYUNAME/userenv && { cd $MY_REPO/build-tools/build-docker-images && ./build-stx-images.sh --os-label=debian --attempts=3 --retry-delay=30 --stream=stable --version=20230904T060000Z --prefix=master --user=starlingx --latest --base=starlingx/stx-debian:master-stable-20230904T060000Z --no-pull-base --wheels=$MY_WORKSPACE/std/build-wheels-debian-stable/stx-debian-stable-wheels.tar --cache --push --os=debian ; }'"'"''

<...>
[2023-09-04T12:41:21.146Z] Step 4/37 : RUN yum -y install gcc-c++ git libtool make python36-devel && pip3 install --no-cache-dir meson==0.57.1 cython grpcio-tools grpcio-reflection pyyaml pyelftools && yum clean all

<...>
[2023-09-04T12:42:19.914Z] Installing collected packages: meson, cython, protobuf, grpcio, grpcio-tools, grpcio-reflection, pyyaml, pyelftools
[2023-09-04T12:42:19.914Z] Running setup.py install for meson: started
[2023-09-04T12:42:20.464Z] Running setup.py install for meson: finished with status 'done'
[2023-09-04T12:42:21.375Z] Running setup.py install for grpcio: started
[2023-09-04T12:43:28.883Z] Running setup.py install for grpcio: still running...
[2023-09-04T12:43:28.883Z] Running setup.py install for grpcio: finished with status 'done'
[2023-09-04T12:43:28.883Z] Running setup.py install for grpcio-tools: started
[2023-09-04T12:43:55.285Z] Running setup.py install for grpcio-tools: finished with status 'done'
[2023-09-04T12:43:55.285Z] Running setup.py install for pyyaml: started
[2023-09-04T12:43:55.285Z] Running setup.py install for pyyaml: finished with status 'error'
[2023-09-04T12:43:55.285Z] Complete output from command /usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-8vkv77ev/pyyaml/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-_095jtnt-record/install-record.txt --single-version-externally-managed --compile:
[2023-09-04T12:43:55.285Z] running install
[2023-09-04T12:43:55.285Z] running build
[2023-09-04T12:43:55.285Z] running build_py
[2023-09-04T12:43:55.285Z] creating build
[2023-09-04T12:43:55.285Z] creating build/lib.linux-x86_64-3.6
[2023-09-04T12:43:55.285Z] creating build/lib.linux-x86_64-3.6/yaml
[2023-09-04T12:43:55.285Z] copying lib/yaml/loader.py -> build/lib.linux-x86_64-3.6/yaml
[2023-09-04T12:43:55.285Z] copying lib/yaml/parser.py -> build/lib.linux-x86_64-3.6/yaml
[2023-09-04T12:43:55.285Z] copying lib/yaml/__init__.py -> build/lib.linux-x86_64-3.6/yaml
[2023-09-04T12:43:55.285Z] copying lib/yaml/resolver.py -> build/lib.linux-x86_64-3.6/yaml
[2023-09-04T12:43:55.285Z] copying lib/yaml/emitter.py -> build/lib.linux-x86_64-3.6/yaml
[2023-09-04T12:43:55.285Z] copying lib/yaml/constructor.py -> build/lib.linux-x86_64-3.6/yaml
[2023-09-04T12:43:55.285Z] copying lib/yaml/representer.py -> build/lib.linux-x86_64-3.6/yaml
[2023-09-04T12:43:55.285Z] copying lib/yaml/nodes.py -> build/lib.linux-x86_64-3.6/yaml
[2023-09-04T12:43:55.285Z] copying lib/yaml/scanner.py -> build/lib.linux-x86_64-3.6/yaml
[2023-09-04T12:43:55.285Z] copying lib/yaml/composer.py -> build/lib.linux-x86_64-3.6/yaml
[2023-09-04T12:43:55.285Z] copying lib/yaml/cyaml.py -> build/lib.linux-x86_64-3.6/yaml
[2023-09-04T12:43:55.286Z] copying lib/yaml/error.py -> build/lib.linux-x86_64-3.6/yaml
[2023-09-04T12:43:55.286Z] copying lib/yaml/dumper.py -> build/lib.linux-x86_64-3.6/yaml
[2023-09-04T12:43:55.286Z] copying lib/yaml/events.py -> build/lib.linux-x86_64-3.6/yaml
[2023-09-04T12:43:55.286Z] copying lib/yaml/reader.py -> build/lib.linux-x86_64-3.6/yaml
[2023-09-04T12:43:55.286Z] copying lib/yaml/serializer.py -> build/lib.linux-x86_64-3.6/yaml
[2023-09-04T12:43:55.286Z] copying lib/yaml/tokens.py -> build/lib.linux-x86_64-3.6/yaml
[2023-09-04T12:43:55.286Z] creating build/lib.linux-x86_64-3.6/_yaml
[2023-09-04T12:43:55.286Z] copying lib/_yaml/__init__.py -> build/lib.linux-x86_64-3.6/_yaml
[2023-09-04T12:43:55.286Z] running build_ext
[2023-09-04T12:43:55.286Z] Traceback (most recent call last):
[2023-09-04T12:43:55.286Z] File "<string>", line 1, in <module>
[2023-09-04T12:43:55.286Z] File "/tmp/pip-build-8vkv77ev/pyyaml/setup.py", line 313, in <module>
[2023-09-04T12:43:55.286Z] python_requires='>=3.6',
[2023-09-04T12:43:55.286Z] File "/usr/lib/python3.6/site-packages/setuptools/__init__.py", line 129, in setup
[2023-09-04T12:43:55.286Z] return distutils.core.setup(**attrs)
[2023-09-04T12:43:55.286Z] File "/usr/lib64/python3.6/distutils/core.py", line 148, in setup
[2023-09-04T12:43:55.286Z] dist.run_commands()
[2023-09-04T12:43:55.286Z] File "/usr/lib64/python3.6/distutils/dist.py", line 955, in run_commands
[2023-09-04T12:43:55.286Z] self.run_command(cmd)
[2023-09-04T12:43:55.286Z] File "/usr/lib64/python3.6/distutils/dist.py", line 974, in run_command
[2023-09-04T12:43:55.286Z] cmd_obj.run()
[2023-09-04T12:43:55.286Z] File "/usr/lib/python3.6/site-packages/setuptools/command/install.py", line 61, in run
[2023-09-04T12:43:55.286Z] return orig.install.run(self)
[2023-09-04T12:43:55.286Z] File "/usr/lib64/python3.6/distutils/command/install.py", line 556, in run
[2023-09-04T12:43:55.286Z] self.run_command('build')
[2023-09-04T12:43:55.286Z] File "/usr/lib64/python3.6/distutils/cmd.py", line 313, in run_command
[2023-09-04T12:43:55.286Z] self.distribution.run_command(command)
[2023-09-04T12:43:55.286Z] File "/usr/lib64/python3.6/distutils/dist.py", line 974, in run_command
[2023-09-04T12:43:55.286Z] cmd_obj.run()
[2023-09-04T12:43:55.286Z] File "/usr/lib64/python3.6/distutils/command/build.py", line 135, in run
[2023-09-04T12:43:55.286Z] self.run_command(cmd_name)
[2023-09-04T12:43:55.286Z] File "/usr/lib64/python3.6/distutils/cmd.py", line 313, in run_command
[2023-09-04T12:43:55.286Z] self.distribution.run_command(command)
[2023-09-04T12:43:55.286Z] File "/usr/lib64/python3.6/distutils/dist.py", line 974, in run_command
[2023-09-04T12:43:55.286Z] cmd_obj.run()
[2023-09-04T12:43:55.286Z] File "/tmp/pip-build-8vkv77ev/pyyaml/setup.py", line 191, in run
[2023-09-04T12:43:55.286Z] _build_ext.run(self)
[2023-09-04T12:43:55.286Z] File "/usr/lib64/python3.6/distutils/command/build_ext.py", line 339, in run
[2023-09-04T12:43:55.286Z] self.build_extensions()
[2023-09-04T12:43:55.286Z] File "/tmp/pip-build-8vkv77ev/pyyaml/setup.py", line 233, in build_extensions
[2023-09-04T12:43:55.286Z] ext.sources = self.cython_sources(ext.sources, ext)
[2023-09-04T12:43:55.286Z] File "/usr/lib64/python3.6/distutils/cmd.py", line 103, in __getattr__
[2023-09-04T12:43:55.286Z] raise AttributeError(attr)
[2023-09-04T12:43:55.286Z] AttributeError: cython_sources
[2023-09-04T12:43:55.286Z]
[2023-09-04T12:43:55.286Z] ----------------------------------------
[2023-09-04T12:43:55.286Z] Command "/usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-8vkv77ev/pyyaml/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-_095jtnt-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-8vkv77ev/pyyaml/
[2023-09-04T12:43:55.286Z] The command '/bin/sh -c yum -y install gcc-c++ git libtool make python36-devel && pip3 install --no-cache-dir meson==0.57.1 cython grpcio-tools grpcio-reflection pyyaml pyelftools && yum clean all' returned a non-zero code: 1

Test Activity
-------------
N/A

Workaround
----------
N/A

Revision history for this message
Davlet Panech (dpanech) wrote :

See also: https://stackoverflow.com/a/76710304

Revision history for this message
Davlet Panech (dpanech) wrote :

Successful logs from a manual build provided by <email address hidden>
* build-uft-image-logs.txt
* build-docker-images.txt

Revision history for this message
Davlet Panech (dpanech) wrote :

Rafal: your build log indicates docker build cache being used:

#22 [stage-0 2/21] RUN yum -y install gcc-c++ git libtool make python36-devel && pip3 install --no-cache-dir meson==0.57.1 cython grpcio-tools grpcio-reflection pyyaml pyelftools && yum clean all
#22 CACHED

Are you sure you didn't add --cache when calling build-stx-images.sh ?

In any case I believe the Docker and/or patches are not buildable in its present state.

Revision history for this message
Davlet Panech (dpanech) wrote :

I disabled this image in the build for the moment: https://review.opendev.org/c/starlingx/app-intel-ethernet-operator/+/893803

Rafal: please be sure to undo it once you have a proper solution.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to app-intel-ethernet-operator (master)

Reviewed: https://review.opendev.org/c/starlingx/app-intel-ethernet-operator/+/893803
Committed: https://opendev.org/starlingx/app-intel-ethernet-operator/commit/0785ca6a97187c09df0174d8891f871f728204f4
Submitter: "Zuul (22348)"
Branch: master

commit 0785ca6a97187c09df0174d8891f871f728204f4
Author: Davlet Panech <email address hidden>
Date: Tue Sep 5 15:10:53 2023 -0400

    Disable the build of uft docker image

    This project is not currently buildable due to [Related-Bug] below. Disable this project to unblock daily builds.

    Related-Bug: #2034255
    Change-Id: I127373a18a3cce1f630b3494c16234b2a22a19dd
    Signed-off-by: Davlet Panech <email address hidden>

Revision history for this message
Rafal Lal (rafal-lal) wrote :

Thanks for your help with this Davlet, seems like building PyYAML 6.0.1 from source was at fault here. Pinning version to 5.4.1 is fixing the issue. As to build-stx-images.sh, I didn't add --cache but I also didn't run it from stx build environment so it somehow used cache from docker present on host. Will create another patch and revert image build block.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to app-intel-ethernet-operator (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/app-intel-ethernet-operator/+/893853

OpenStack Infra (hudson-openstack)
Changed in starlingx:
status: New → In Progress
Ghada Khalil (gkhalil)
tags: added: stx.9.0 stx.build
Changed in starlingx:
assignee: nobody → Rafal Lal (rafal-lal)
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to app-intel-ethernet-operator (master)

Reviewed: https://review.opendev.org/c/starlingx/app-intel-ethernet-operator/+/893853
Committed: https://opendev.org/starlingx/app-intel-ethernet-operator/commit/a97632facbd74c30d19cd97c79f79cd7be68d6a3
Submitter: "Zuul (22348)"
Branch: master

commit a97632facbd74c30d19cd97c79f79cd7be68d6a3
Author: Rafal Lal <email address hidden>
Date: Wed Sep 6 11:27:26 2023 +0200

    Enable build of uft docker image

    - Add patch that fixes PyYAML build error during UFT image build
    - Rename existing patch to fit naming convention

    Closes-Bug: #2034255
    Change-Id: I9a8b1e781e88d0fd0e2c38b6a64c74cb653b7d4e
    Signed-off-by: Rafal Lal <email address hidden>

Changed in starlingx:
status: In Progress → Fix Released
Revision history for this message
Davlet Panech (dpanech) wrote :

Re-opening bug. Latest build failed to build this image (but didn't report the failure to Jenkins, so the overall build looks "successful").

Search for "uft" here: https://mirror.starlingx.cengn.ca/mirror/starlingx/master/debian/monolithic/20230908T060000Z/logs/build-docker-images-stable.log.txt

[2023-09-08T12:44:11.336Z] Building uft
[2023-09-08T12:44:11.336Z] get_git 'https://github.com/intel/UFT.git' '43cdc7277897093f7e894bfc103e7ca1fe9edc8c' '/localdisk/designer/jenkins/debian-master/cgcs-root/stx/app-intel-ethernet-operator/uft-image/debian/../files/uft-fix.patch'
[2023-09-08T12:44:11.586Z] Running: /bin/bash -c rm -rf UFT.clone_tmp && git clone --recursive https://github.com/intel/UFT.git UFT.clone_tmp
[2023-09-08T12:44:11.586Z] Cloning into 'UFT.clone_tmp'...
[2023-09-08T12:44:12.137Z] Note: switching to '43cdc7277897093f7e894bfc103e7ca1fe9edc8c'.
[2023-09-08T12:44:12.137Z]
[2023-09-08T12:44:12.137Z] You are in 'detached HEAD' state. You can look around, make experimental
[2023-09-08T12:44:12.137Z] changes and commit them, and you can discard any commits you make in this
[2023-09-08T12:44:12.137Z] state without impacting any branches by switching back to a branch.
[2023-09-08T12:44:12.137Z]
[2023-09-08T12:44:12.137Z] If you want to create a new branch to retain commits you create, you may
[2023-09-08T12:44:12.137Z] do so (now or later) by using -c with the switch command. Example:
[2023-09-08T12:44:12.137Z]
[2023-09-08T12:44:12.137Z] git switch -c <new-branch-name>
[2023-09-08T12:44:12.137Z]
[2023-09-08T12:44:12.137Z] Or undo this operation with:
[2023-09-08T12:44:12.137Z]
[2023-09-08T12:44:12.137Z] git switch -
[2023-09-08T12:44:12.137Z]
[2023-09-08T12:44:12.137Z] Turn off this advice by setting config variable advice.detachedHead to false
[2023-09-08T12:44:12.137Z]
[2023-09-08T12:44:12.137Z] HEAD is now at 43cdc72 Merge pull request #5 from yiding-zhou/main
[2023-09-08T12:44:12.137Z] fatal: could not open '/localdisk/designer/jenkins/debian-master/cgcs-root/stx/app-intel-ethernet-operator/uft-image/debian/../files/uft-fix.patch' for reading: No such file or directory
[2023-09-08T12:44:12.137Z] Failed to apply /localdisk/designer/jenkins/debian-master/cgcs-root/stx/app-intel-ethernet-operator/uft-image/debian/../files/uft-fix.patch in UFT
[2023-09-08T12:44:12.137Z] Aborting...
[2023-09-08T12:44:12.137Z] Failed to clone or update https://github.com/intel/UFT.git. Aborting...
[2023-09-08T12:44:12.137Z] Building power-node-agent

Changed in starlingx:
status: Fix Released → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to root (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/root/+/894388

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to app-intel-ethernet-operator (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/app-intel-ethernet-operator/+/894506

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to app-intel-ethernet-operator (master)

Reviewed: https://review.opendev.org/c/starlingx/app-intel-ethernet-operator/+/894506
Committed: https://opendev.org/starlingx/app-intel-ethernet-operator/commit/e421dea1c4d0a4dfebdcc8b48718491780f99883
Submitter: "Zuul (22348)"
Branch: master

commit e421dea1c4d0a4dfebdcc8b48718491780f99883
Author: Rafal Lal <email address hidden>
Date: Mon Sep 11 10:42:30 2023 +0200

    Fix uft image build error

    - Correct patch names
    - Correct 0002 patch so it applies correctly

    Related-Bug: #2034255
    Change-Id: Ieb8116aa8b77b4d988745a7295db8879362cc54d
    Signed-off-by: Rafal Lal <email address hidden>

Revision history for this message
Davlet Panech (dpanech) wrote :

It compile now, but uft's build script doesn't assign the correct tag to the image it generates, causing the build to fail:

https://mirror.starlingx.cengn.ca/mirror/starlingx/master/debian/monolithic/20230912T060000Z/logs/build-docker-images-stable.log.txt

[2023-09-12T12:41:55.685Z] Error response from daemon: No such image: jenkins/uft:master-debian-stable-build
[2023-09-12T12:41:55.685Z] Running: docker push starlingx/uft:master-debian-stable-20230912T060000Z.0
[2023-09-12T12:41:55.685Z] An image does not exist locally with the tag: starlingx/uft
[2023-09-12T12:41:55.685Z] The push refers to repository [docker.io/starlingx/uft]
[2023-09-12T12:41:55.685Z] Command (docker) failed, attempt 1 of 3.

=========

I'll try to reproduce it and propose a fix in a bit

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to root (master)

Reviewed: https://review.opendev.org/c/starlingx/root/+/894388
Committed: https://opendev.org/starlingx/root/commit/d33c5b7120ec853a98ca06c15a2bd1b2a8f7c517
Submitter: "Zuul (22348)"
Branch: master

commit d33c5b7120ec853a98ca06c15a2bd1b2a8f7c517
Author: Davlet Panech <email address hidden>
Date: Fri Sep 8 12:58:52 2023 -0400

    build-docker-images: additional error handling

    Handle errors caused by failed git clone attempts.

    Related-Bug: 2034255
    Signed-off-by: Davlet Panech <email address hidden>
    Change-Id: Ie3708a6522ac4e4ebda9502b697de30927b3d5eb

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to app-intel-ethernet-operator (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/app-intel-ethernet-operator/+/894819

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to app-intel-ethernet-operator (master)

Reviewed: https://review.opendev.org/c/starlingx/app-intel-ethernet-operator/+/894819
Committed: https://opendev.org/starlingx/app-intel-ethernet-operator/commit/63b6de62b5f0e5631b968cdac29cd3590f83704a
Submitter: "Zuul (22348)"
Branch: master

commit 63b6de62b5f0e5631b968cdac29cd3590f83704a
Author: Davlet Panech <email address hidden>
Date: Tue Sep 12 12:15:56 2023 -0400

    uft: set docker image tag correctly

    Docker image custom build script is always passed the expected image
    name/tag as the last argument by the master build script. This project
    was ignoring that argument and always generated an image called
    "uft:latest".

    Solution: use last command-line argument as the image name/tag to be
    generated.

    TESTS
    ======================
    * Build & push uft image via Jenkins

    Closes-Bug: 2034255
    Change-Id: Ieeea76ae9dc9b47a2c6c6875a66e0b3879baf497
    Signed-off-by: Davlet Panech <email address hidden>

Changed in starlingx:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.