newton: tripleo-quickstart jobs don't build packages from stable/newton

[Michele Baldessari]

In backporting https://review.openstack.org/#/c/447229/ to stable/newton I consistently see the following CI error:
http://logs.openstack.org/29/447229/2/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/c1f6ba4/logs/quickstart_install.txt.gz

TASK [build-test-packages : Run DLRN] ******************************************
task path: /home/jenkins/workspace/gate-tripleo-ci-centos-7-nonha-multinode-oooq/.quickstart/usr/local/share/ansible/roles/build-test-packages/tasks/dlrn-build.yml:53
Saturday 18 March 2017 22:01:04 +0000 (0:00:01.182) 0:04:16.893 ********
fatal: [undercloud]: FAILED! => {"changed": true, "cmd": "set +e; source /home/jenkins/dlrn-venv/bin/activate; while true; do\n dlrn --config-file projects.ini --head-only --package-name openstack-tripleo-heat-templates --local --info-repo rdoinfo --dev;\n if [ $? -eq 0 ]; then\n # SUCCESS\n break;\n elif [ $? -eq 1 ]; then\n # FAILED\n exit 1;\n elif [ $? -eq 2 ]; then\n # RETRY\n continue;\n fi;\n # Unexpected DLRN return code\n exit $?;\n done;", "delta": "0:02:02.897140", "end": "2017-03-18 22:03:08.844269", "failed": true, "rc": 1, "start": "2017-03-18 22:01:05.947129", "stderr": "2017-03-18 22:01:06,817 INFO:dlrn-repositories:Getting https://github.com/rdo-packages/tripleo-heat-templates-distgit.git to ./data/openstack-tripleo-heat-templates_distro (rpm-newton)\n2017-03-18 22:01:07,895 INFO:dlrn-repositories:Falling back to rpm-master\n2017-03-18 22:01:08,206 INFO:dlrn-repositories:Getting git://git.openstack.org/openstack/tripleo-heat-templates to ./data/openstack-tripleo-heat-templates (stable/newton)\n2017-03-18 22:01:08,504 INFO:dlrn-build:Processing openstack-tripleo-heat-templates 50f0b974c8fd764c8a0cac5124d28e8382a73658\n2017-03-18 22:03:07,138 ERROR:dlrn-build:cmd failed. See logs at: /home/jenkins/DLRN/data/repos/50/f0/50f0b974c8fd764c8a0cac5124d28e8382a73658_dev/\n2017-03-18 22:03:07,139 INFO:dlrn:Received exception Error in build_rpm_wrapper for openstack-tripleo-heat-templates: \n\n RAN: /usr/bin/env /home/jenkins/DLRN/scripts/build_rpm.sh centos openstack-tripleo-heat-templates /home/jenkins/DLRN/data/repos/50/f0/50f0b974c8fd764c8a0cac5124d28e8382a73658_dev /home/jenkins/DLRN/data https://trunk.rdoproject.org/centos7-newton /home/jenkins/DLRN/data/openstack-tripleo-heat-templates_distro\n\n STDOUT:\n\n\n STDERR:\n\n2017-03-18 22:03:07,144 ERROR:dlrn:Received exception Error in build_rpm_wrapper for openstack-tripleo-heat-templates: \n\n RAN: /usr/bin/env /home/jenkins/DLRN/scripts/build_rpm.sh centos openstack-tripleo-heat-templates /home/jenkins/DLRN/data/repos/50/f0/50f0b974c8fd764c8a0cac5124d28e8382a73658_dev /home/jenkins/DLRN/data https://trunk.rdoproject.org/centos7-newton /home/jenkins/DLRN/data/openstack-tripleo-heat-templates_distro\n\n STDOUT:\n\n\n STDERR:\n\n2017-03-18 22:03:07,149 INFO:dlrn-notifications:Skipping notify email to ['<email address hidden>', '<email address hidden>']", "stdout": "", "stdout_lines": [], "warnings": []}

The error is:
http://logs.openstack.org/29/447229/2/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/c1f6ba4/logs/delorean_logs/50/f0/50f0b974c8fd764c8a0cac5124d28e8382a73658_dev/build.log.txt.gz

+ cp -ar scripts /builddir/build/BUILDROOT/openstack-tripleo-heat-templates-5.2.1-0.20170318220252.50f0b97.el7.centos.x86_64//usr/share/openstack-tripleo-heat-templates
cp: cannot stat 'scripts': No such file or directory
RPM build errors:
error: Bad exit status from /var/tmp/rpm-tmp.EcBM03 (%install)
    Bad exit status from /var/tmp/rpm-tmp.EcBM03 (%install)
Child return code was: 1
EXCEPTION: [Error()]
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 88, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.7/site-packages/mockbuild/util.py", line 547, in do
    raise exception.Error("Command failed. See logs for output.\n # %s" % (command,), child.returncode)
Error: Command failed. See logs for output.
 # bash --login -c /usr/bin/rpmbuild -bb --target x86_64 --nodeps /builddir/build/SPECS/openstack-tripleo-heat-templates.spec

[Michele Baldessari (michele)]

Reproducer:
mkdir -p /tmp/test && cd /tmp/test
git clone https://github.com/openstack-infra/tripleo-ci
cd tripleo-ci/
TRIPLEO_ROOT=/tmp/test ./scripts/tripleo.sh --delorean-setup
STABLE_RELEASE=newton TRIPLEO_ROOT=/tmp/test ./scripts/tripleo.sh --delorean-build openstack/tripleo-heat-templates

[Michele Baldessari (michele)]

No idea what is wrong because https://github.com/rdo-packages/tripleo-heat-templates-distgit/blob/newton-rdo/openstack-tripleo-heat-templates.spec does not have any mention of 'cp scripts', so we must be pulling some wrong branch

[Michele Baldessari (michele)]

I wonder if the issue is actually here: https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/build-test-packages/tasks/dlrn-build.yml#L2-L6 we set the branch for rdo to 'rpm-{stablebranchname}', but I don't see any rpm-newton branch here https://github.com/rdo-packages/tripleo-heat-templates-distgit/blob/newton-rdo/openstack-tripleo-heat-templates.spec only one called 'newton-rdo', so maybe that is the root of the problem?

[Alan Pevec (apevec)]

Michele, you're right, this was fixed in tripleo.sh by https://review.openstack.org/399578
and followups: https://review.openstack.org/424637 and https://review.openstack.org/441960

[Ben Kero (ben-kero)]

I have proposed a change to tripleo-quickstart-extras build-test-packages role similar to the tripleo.sh fix in this change: https://review.openstack.org/399578

In my local testing this results in a newton package of tripleo-quickstart-extras being built successfully.

You can see my change here: https://review.openstack.org/#/c/447747/

As an explanation, see comment 3. Michele is correct that the branch name of the package repository changed for the newton release to the format 'newton-rdo' from 'rpm-mitaka'. See the branches listed here for an example: https://github.com/rdo-packages/tripleo-heat-templates-distgit

The change adds a conditional to check for the newton release and sets the branch name to the appropriate value.

I don't have an explanation of why this isn't happening on ocata though, as that also is using the new 'ocata-rdo' branch name.

[Attila Darazs (adarazs)]

Sorry folks, there is supposed to be a Gerrit - Launchpad integration that seems to have failed this time.

Here's the correct fix, submitted just half an hour after I took ownership of the bug:

https://review.openstack.org/447530

I added the "Closes-Bug", so we should look into why this integration doesn't work.

@Ben: you can abandon your change, looks similar to mine but it doesn't handle the newer releases that come after newton (only master and mitaka is using the rpm-$VERSION format now, and every stable branch going forward is $VERSION-rdo)

I'll submit the change as soon as it passes gates. Gates were so backed up yesterday that I couldn't submit it before going to sleep (the last gate job finished 8 hours after I submitted the change).

[Michele Baldessari (michele)]

Since https://review.openstack.org/#/c/447530/ merged I will close this. Thanks everyone

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-quickstart-extras 2.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-quickstart-extras 2.1.1 release.