[DIB] Scala website changed and hadoop-mapr fails

[Luigi Toscano]

http://www.scala-lang.org changed its structure so the crude code which accessed it and read the version does not work anymore.

At least the code which find the version should be adapted.

Bonus point for updating the download links from download.typesafe.com to download.lightbend.com.

This is a candidate for backport on all supported branches.

[Luigi Toscano (ltoscano)]

Confirmed by the gates, where the gate-sahara-image-elements-buildimages-mapr-ubuntu-xenial-nv job is failing:

2017-07-14 08:29:57.407363 | + echo 'START: installing Scala'
2017-07-14 08:29:57.407375 | START: installing Scala
2017-07-14 08:29:57.407386 | + DEF_VERSION=2.11.6
2017-07-14 08:29:57.407491 | ++ curl -s -o /dev/null -w '%{http_code}' http://www.scala-lang.org/
2017-07-14 08:29:59.900495 | + RETURN_CODE=200
2017-07-14 08:29:59.900559 | + '[' 200 '!=' 200 ']'
2017-07-14 08:29:59.900577 | + '[' trusty == GenericCloud ']'
2017-07-14 08:29:59.900865 | ++ curl -s --fail http://www.scala-lang.org
2017-07-14 08:29:59.901051 | ++ grep scala-version
2017-07-14 08:29:59.901220 | ++ grep -Eo '([0-9]\.?)+'
2017-07-14 08:30:00.416949 | + VERSION=
2017-07-14 08:30:00.728476 | 2017-07-14 08:30:00.728 DEBUG diskimage_builder.block_device.blockdevice [-] Creating BlockDevice object __init__ /home/jenkins/workspace/gate-sahara-image-elements-buildimages-mapr-ubuntu-xenial-nv/.tox/venv/local/lib/python2.7/site-packages/diskimage_builder/block_device/blockdevice.py:210
2017-07-14 08:30:00.728791 | 2017-07-14 08:30:00.728 DEBUG diskimage_builder.block_device.blockdevice [-] Params [{'config': '/tmp/dib_build.QU2veePD/block-device/config.yaml', 'mount-base': '/tmp/dib_build.QU2veePD/mnt', 'build-dir': '/tmp/dib_build.QU2veePD', 'root-fs-type': 'ext4', 'root-label': None, 'image-dir': '/tmp/dib_image.TEWPnWnG'}] __init__ /home/jenkins/workspace/gate-sahara-image-elements-buildimages-mapr-ubuntu-xenial-nv/.tox/venv/local/lib/python2.7/site-packages/diskimage_builder/block_device/blockdevice.py:213
2017-07-14 08:30:00.729145 | 2017-07-14 08:30:00.729 INFO diskimage_builder.block_device.blockdevice [-] State already cleaned - no way to do anything here

[Luigi Toscano (ltoscano)]

Uhm, something fishy; there is a condition which should be triggered when the the version can't be retrieved from the website. Requires some more testing.

This happens only on CentOS 7.

[Luigi Toscano (ltoscano)]

And in fact it triggered successfully from another job:
http://logs.openstack.org/51/483251/2/check/gate-sahara-image-elements-buildimages-mapr-ubuntu-xenial-nv/a3ca4c9/console.html

So maybe a temporary error?

[Luigi Toscano (ltoscano)]

Even if the failed build could have other issues, the structure of the page changed changes slightly between July 8th and now (thanks Internet Archive) and the code that scrapes the page should be adapted, or the old version (2.11.x) would be always used.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/483925
Committed: https://git.openstack.org/cgit/openstack/sahara-image-elements/commit/?id=83c3011f69389e5e1459e5005134a0ff52e8bfcd
Submitter: Jenkins
Branch: master

commit 83c3011f69389e5e1459e5005134a0ff52e8bfcd
Author: Luigi Toscano <email address hidden>
Date: Fri Jul 14 18:12:37 2017 +0200

    mapr: fix the discovery of the version of Scala

    The structure of the page changed slightly between July 8th
    and now (thanks Internet Archive).
    Adapt the extraction code. including a more robust regex.

    Also update the download location (lightbend.com instead of
    typesafe.com, even if the latter is still a redirect).

    Finally, expose a new variable which allows users to
    force a specific version of Scala, disabling the autodetection.
    This is useful to workaround possible future similar breakages
    without changing the code.

    Closes-Bug: #1704337

    Change-Id: Ib105d08816d6833f05301e59f64163de5f070109

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/484021

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/484025

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/484021
Committed: https://git.openstack.org/cgit/openstack/sahara-image-elements/commit/?id=e8c4064bcc83454eabc5906f71ea6fcd3f1e03ee
Submitter: Jenkins
Branch: stable/ocata

commit e8c4064bcc83454eabc5906f71ea6fcd3f1e03ee
Author: Luigi Toscano <email address hidden>
Date: Fri Jul 14 18:12:37 2017 +0200

    mapr: fix the discovery of the version of Scala

    The structure of the page changed slightly between July 8th
    and now (thanks Internet Archive).
    Adapt the extraction code. including a more robust regex.

    Also update the download location (lightbend.com instead of
    typesafe.com, even if the latter is still a redirect).

    Finally, expose a new variable which allows users to
    force a specific version of Scala, disabling the autodetection.
    This is useful to workaround possible future similar breakages
    without changing the code.

    Closes-Bug: #1704337

    Change-Id: Ib105d08816d6833f05301e59f64163de5f070109
    (cherry picked from commit 83c3011f69389e5e1459e5005134a0ff52e8bfcd)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/484025
Committed: https://git.openstack.org/cgit/openstack/sahara-image-elements/commit/?id=049c2e8d3ef76ac9ddae08f619a7899f9726007a
Submitter: Jenkins
Branch: stable/newton

commit 049c2e8d3ef76ac9ddae08f619a7899f9726007a
Author: Luigi Toscano <email address hidden>
Date: Fri Jul 14 18:12:37 2017 +0200

    mapr: fix the discovery of the version of Scala

    The structure of the page changed slightly between July 8th
    and now (thanks Internet Archive).
    Adapt the extraction code. including a more robust regex.

    Also update the download location (lightbend.com instead of
    typesafe.com, even if the latter is still a redirect).

    Finally, expose a new variable which allows users to
    force a specific version of Scala, disabling the autodetection.
    This is useful to workaround possible future similar breakages
    without changing the code.

    Closes-Bug: #1704337

    Change-Id: Ib105d08816d6833f05301e59f64163de5f070109
    (cherry picked from commit 83c3011f69389e5e1459e5005134a0ff52e8bfcd)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/sahara-image-elements 7.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/sahara-image-elements 5.0.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/sahara-image-elements 6.0.2 release.