charm fails to build

[Ryan Beisner]

All of the reactive openstack charms are failing to build at this time.

00:01:41.420 Collecting pyaml (from charms.reactive<2.0.0,>=0.1.0->-r /var/lib/jenkins/checkout/34/designate/build/deps/layer/basic/wheelhouse.txt (line 3))
00:01:41.420 Using cached pyaml-17.10.0.tar.gz
00:01:41.420 Saved /tmp/tmpbpYLc9/pyaml-17.10.0.tar.gz
00:01:41.420 Collecting MarkupSafe>=0.23 (from Jinja2->charmhelpers<1.0.0,>=0.4.0->-r /var/lib/jenkins/checkout/34/designate/build/deps/layer/basic/wheelhouse.txt (line 2))
00:01:41.420 Using cached MarkupSafe-1.0.tar.gz
00:01:41.420 Saved /tmp/tmpbpYLc9/MarkupSafe-1.0.tar.gz
00:01:41.420 Complete output from command python setup.py egg_info:
00:01:41.420 Traceback (most recent call last):
00:01:41.420 File "<string>", line 1, in <module>
00:01:41.420 File "/tmp/tmpOL0VjZ/lib/python3.5/tokenize.py", line 454, in open
00:01:41.420 buffer = _builtin_open(filename, 'rb')
00:01:41.420 FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-build-7rgzaij3/MarkupSafe/setup.py'
00:01:41.420
00:01:41.420 ----------------------------------------
00:01:41.420 Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-7rgzaij3/MarkupSafe/
00:01:41.420
00:01:41.420 Command failed: bash -c . /tmp/tmpOL0VjZ/bin/activate ; pip3 download --no-binary :all: -d /tmp/tmpbpYLc9 -r /var/lib/jenkins/checkout/34/designate/build/deps/layer/basic/wheelhouse.txt
00:01:41.514 ERROR: InvocationError: '/var/lib/jenkins/checkout/34/designate/.tox/build/bin/charm-build --log-level DEBUG -o /var/lib/jenkins/checkout/34/designate/build src'

[Ryan Beisner (1chb1n)]

Once confirmed, please also add other charms to the bug tasks.

[Andrew McLeod (admcleod)]

I am not able to replicate this: manual build and tox -e build both work. I've retriggered the build job in the CI system, and it passed successfully.

[Andrew McLeod (admcleod)]

Further investigation reveals that these will rebuild individually, but not simultaneously, indicating an infrastructure issue.

[Frode Nordahl (fnordahl)]

I often see random build failures, and your observation about this being more likely to happen during concurrent builds seems plausible. It was certainly the case for these failures as they occurred just after posting a change to rebuild all the reactive charms.

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_smoke/openstack/charm-octavia/677889/1/11754/consoleText.charm_build_30946.txt

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_smoke/openstack/charm-octavia-diskimage-retrofit/677883/1/11748/consoleText.charm_build_30940.txt

Could it be that multiple build jobs are executed on the same task runner, and that they share a common pip cache or temporary directory?

[Frode Nordahl (fnordahl)]

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_smoke/openstack/charm-octavia/751663/1/19123/consoleText.charm_build_35586.txt

[Frode Nordahl (fnordahl)]

This look a bit suspicious to me:

    https://bazaar.launchpad.net/~uosci/ubuntu-openstack-ci/trunk/view/head:/run/job-parts/osci_job_common.sh#L477

The mtime of the files pip extract in /tmp will almost always be older than one hour, it is set from the extracted archive.