Undercloud / Overcloud Heat stack fails on: YAQL list index out of range (includes upgrades cases)

[Dan Prince]

I rebased my undercloud-containers setup today (on a fresh VM) and found my stack creations were failing. Specifically the UndercloudHostsDeployment resource was failing in the heat stack.

This YAQL expression was failing:

dict($.data.where($ != null).flatten().selectMany($.items()).groupBy($[0], $[1], [$[0], $[1].flatten()]))

No recent changes in t-h-t around how we construct this expression or the data that gets passed in. On a hunch I downgraded the python yaql package and was able to resolve the issue.

---------

Removed:
  python2-yaql.noarch 0:1.1.3-2.el7

Installed:
  python2-yaql.noarch 0:1.1.0-3.el7

[Emilien Macchi (emilienm)]

I confirm, see http://logs.openstack.org/06/542906/31/check/tripleo-ci-centos-7-undercloud-containers/8d29ce8/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-02-16_23_04_20

[Harald Jensås (harald-jensas)]

Overcloud deploy fails with similar error. Ref: https://bugs.launchpad.net/tripleo/+bug/1749977

Probably same issue.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/545498

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/545498
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=e897da3b690dd69883483d818e957fca85721622
Submitter: Zuul
Branch: master

commit e897da3b690dd69883483d818e957fca85721622
Author: Emilien Macchi <email address hidden>
Date: Fri Feb 16 17:27:24 2018 -0800

    Update YAQL queries with groupBy

    YAQL introduced a backward incompatible change in one of its minor
    versions:
    https://github.com/openstack/yaql/commit/3fb91784018de335440b01b3b069fe45dc53e025#diff-f36776b660e5fe4f88e3295e5b751396R215

    It changes the expected behavior of groupBy() aggregator, so we need to
    update our queries otherwise it fails with a "list index out of range"
    error.

    Change-Id: I2ca2ebb2c8d22aeedbcb6920072db5b6dba3311b
    Closes-Bug: #1750032
    Co-Authored-By: Alex Schultz <email address hidden>

[Dan Prince (dan-prince)]

Would it be better for all projects using YAQL to revert the breaking change there instead. It seems like we shouldn't change how YAQL functions work and expect all projects to upgrade to this.

Also note the error here was really cryptic. I only pin-pointed the YAQL package by going in and adding LOG.debug statements to heat. I think several people were confused at the root cause of the issue because of the lack of logging around this...

[Dan Prince (dan-prince)]

sigh, it looks like that change was from 2016. Does it really take us a year to publish a new YAQL package?

[Alan Pevec (apevec)]

It was treated as a dependency in RDO rebuilt from Fedora and there were no updates there until update was pushed by Alfredo recently https://src.fedoraproject.org/rpms/python-yaql/pull-request/1
I'd suggest team(s) depending on yaql to take over ownership in Fedora and RDO.

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

This needs backport:
 - for testing upgrade in mixed upgrade setup: In that case the undercloud is in queen with the new yaql and the overcloud fails to deploy[1]

 - support of operations on the overcloud after an upgrade of the overcloud.

[1] See this log with index failure: http://logs.openstack.org/62/545762/3/experimental/tripleo-ci-centos-7-scenario001-multinode-oc-upgrade/afc98a5/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-02-19_09_48_54

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/545817

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

But then the original deployment will certainly fails ... not sure how to deal with both cases... Put the backport as WIP.

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

Note, that it affect also mixed version for fast forward upgrade[1] (master undercloud/newton overcloud deployment)

[1] http://logs.openstack.org/86/525686/55/experimental/tripleo-ci-centos-7-scenario001-multinode-ffu-upgrade/5412555/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

[Bogdan Dobrelya (bogdando)]

Shall we re-open it and put alert back then?

[Bogdan Dobrelya (bogdando)]

Cuz of remaining upgrades impact, I lowered it to high and re-opened with a new tag added. Hope that works and we need no to open another bug for the same issue.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/545856

[Zane Bitter (zaneb)]

Proposed fix to YAQL itself for backwards compatibility: https://review.openstack.org/#/c/545996/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/545856
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=0ddfff79ee5a5f62bb6de55b873bd16b811905f3
Submitter: Zuul
Branch: master

commit 0ddfff79ee5a5f62bb6de55b873bd16b811905f3
Author: Thomas Herve <email address hidden>
Date: Mon Feb 19 16:32:42 2018 +0100

    Do not use the 3rd argument of yaql groupBy

    Due to an incompatible change in yaql, it's hard to use the aggregration
    of groupBy as the behavior is completely different depending on the
    version. Let's try to not rely on it.

    Change-Id: I2887011f6baf4867d422579b116b5e143acf5679
    Related-Bug: #1750032

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/546094

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/546113

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/546118

[OpenStack Infra (hudson-openstack)]

Change abandoned by Athlan-Guyot sofer (<email address hidden>) on branch: stable/pike
Review: https://review.openstack.org/545817
Reason: In favor of https://review.openstack.org/#/q/I2887011f6baf4867d422579b116b5e143acf5679

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/546094
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=677c4c221696b16d10ac80709af4be49e6e334c2
Submitter: Zuul
Branch: stable/pike

commit 677c4c221696b16d10ac80709af4be49e6e334c2
Author: Thomas Herve <email address hidden>
Date: Mon Feb 19 16:32:42 2018 +0100

    Do not use the 3rd argument of yaql groupBy

    Due to an incompatible change in yaql, it's hard to use the aggregration
    of groupBy as the behavior is completely different depending on the
    version. Let's try to not rely on it.

    Change-Id: I2887011f6baf4867d422579b116b5e143acf5679
    Related-Bug: #1750032
    (cherry picked from commit 0ddfff79ee5a5f62bb6de55b873bd16b811905f3)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/546113
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=dcd8a695bedad86f111cf38d204c84ce8d5bfb62
Submitter: Zuul
Branch: stable/ocata

commit dcd8a695bedad86f111cf38d204c84ce8d5bfb62
Author: Thomas Herve <email address hidden>
Date: Mon Feb 19 16:32:42 2018 +0100

    Do not use the 3rd argument of yaql groupBy

    Due to an incompatible change in yaql, it's hard to use the aggregration
    of groupBy as the behavior is completely different depending on the
    version. Let's try to not rely on it.

    Change-Id: I2887011f6baf4867d422579b116b5e143acf5679
    Related-Bug: #1750032
    (cherry picked from commit 0ddfff79ee5a5f62bb6de55b873bd16b811905f3)
    (cherry picked from commit 677c4c221696b16d10ac80709af4be49e6e334c2)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/546118
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=997475c9f545f2b2cee3647c75567d07699da3b4
Submitter: Zuul
Branch: stable/newton

commit 997475c9f545f2b2cee3647c75567d07699da3b4
Author: Thomas Herve <email address hidden>
Date: Mon Feb 19 16:32:42 2018 +0100

    Do not use the 3rd argument of yaql groupBy

    Due to an incompatible change in yaql, it's hard to use the aggregration
    of groupBy as the behavior is completely different depending on the
    version. Let's try to not rely on it.

    Change-Id: I2887011f6baf4867d422579b116b5e143acf5679
    Related-Bug: #1750032
    (cherry picked from commit 0ddfff79ee5a5f62bb6de55b873bd16b811905f3)
    (cherry picked from commit 677c4c221696b16d10ac80709af4be49e6e334c2)
    (cherry picked from commit dcd8a695bedad86f111cf38d204c84ce8d5bfb62)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 8.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/545996
Committed: https://git.openstack.org/cgit/openstack/yaql/commit/?id=74cb81b28053ebe41e1e0031cfcb30f3039f0857
Submitter: Zuul
Branch: master

commit 74cb81b28053ebe41e1e0031cfcb30f3039f0857
Author: Zane Bitter <email address hidden>
Date: Mon Feb 19 17:00:22 2018 -0500

    Make changes to groupBy() backward-compatible

    The behaviour of the groupBy() function was fixed in 1.1.2 by
    3fb91784018de335440b01b3b069fe45dc53e025 to pass only the list of values
    to be aggregated to the aggregator function, instead of passing both the
    key and the list of values (and expecting both the key and the
    aggregated value to be returned). This fix was incompatible with
    existing expressions that used the aggregator argument to groupBy().

    In the event of an error, fall back trying the previous syntax and see
    if something plausible gets returned. If not, re-raise the original error.
    This should mean that pre-existing expressions will continue to work,
    while most outright bogus expressions should fail in a manner consistent
    with the correct definition of the function.

    Change-Id: Ic6c54be4ed99003fe56cf1a5329f3f1d84fd43c8
    Closes-Bug: #1750032

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/yaql 2.0.0 release.