devstack jobs failing zuul during glance DB initialization

[Al Bailey]

STX config devstack zuul job is failing when it sets up glance.

This is blocking all commits to that repo. it will also block all cherry-picks to the R2.0 branch.

Here is the sample Zuul output

File "/opt/stack/glance/glance/db/sqlalchemy/alembic_migrations/data_migrations/train_migrate01_backend_to_store.py", line 28, in has_migrations
    metadata_backend = con.execute(sql_query)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 982, in execute
    return self._execute_text(object_, multiparams, params)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1155, in _execute_text
    parameters,
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
    e, statement, parameters, cursor, context
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1464, in _handle_dbapi_exception
    util.raise_from_cause(newraise, exc_info)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 552, in do_execute
    cursor.execute(statement, parameters)
DBError: (psycopg2.errors.UndefinedFunction) function instr(text, unknown) does not exist
LINE 1: select meta_data from image_locations where INSTR(meta_data,...
                                                    ^
HINT: No function matches the given name and argument types. You might need to add explicit type casts.

[SQL: select meta_data from image_locations where INSTR(meta_data, '"backend":') > 0]
(Background on this error at: http://sqlalche.me/e/f405)

This review recently merged which does not seem to work with our
devstack configuration.
https://review.opendev.org/#/c/665606/

[Al Bailey (albailey1974)]

The attempt at just removing nova/neutron from the requirements did not work.

New Review is up which removes the devstack job from zuul altogether
https://review.opendev.org/#/c/676719/

[Ghada Khalil (gkhalil)]

Marking as stx.2.0 / critical given that this is preventing code merges to stx-config

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/676719
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=edec811bef0eb4e8d7dd9c4b95f7320f41560205
Submitter: Zuul
Branch: master

commit edec811bef0eb4e8d7dd9c4b95f7320f41560205
Author: Al Bailey <email address hidden>
Date: Thu Aug 15 10:28:11 2019 -0500

    Remove devstack from zuul for config due to glance issues

    It looks like devstack zuul is failing when it sets up glance.

    This review recently merged which does not seem to work with our
    devstack configuration.
    https://review.opendev.org/#/c/665606/

    Removing this devstack job from zuul. It can be re-enabled if
    the compatability issues between our environment and glance
    are resolved.

    Change-Id: I0da7fbbef6a6ab65b5146435c6e01fff347c53d1
    Closes-Bug: 1840292
    Signed-off-by: Al Bailey <email address hidden>

[Al Bailey (albailey1974)]

I believe that these 3 repos are also broken, because they also enable glance because their devstack zuul job has a parent flock-devstack-base.
 - ha
 - metal
 - integ

However other repos (like fault) are not broken because they have a parent devstack zuul job of
flock-devstack-base-min

I will convert those other 3 repos to be flock-devstack-base-min.
If we decide to re-enable config, it should also be converted to flock-devstack-base-min.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/676752

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/676753

[Al Bailey (albailey1974)]

integ is not affected.

HA review is here: https://review.opendev.org/#/c/676752/

Metal review is here: https://review.opendev.org/#/c/676753/

[Al Bailey (albailey1974)]

It looks like these will not need to be cherry picked to the R2 branch.
Commits in the R2 branch are skipping their devstack zuul jobs already, due to something in how the job or role has been declared.

[Ghada Khalil (gkhalil)]

Removing the stx.2.0 label based on Al's note above that this issue doesn't impact the r/stx.2.0 release branch

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/676752
Committed: https://git.openstack.org/cgit/starlingx/ha/commit/?id=d7dc7b1eaaf8c2bed7717d37f1c3ee170abdbece
Submitter: Zuul
Branch: master

commit d7dc7b1eaaf8c2bed7717d37f1c3ee170abdbece
Author: Al Bailey <email address hidden>
Date: Thu Aug 15 13:43:27 2019 -0500

    Fixing failing devstack zuul job

    The glance devstack plugin is not working for us,
    and is not needed for our devstack to work, so updating
    the zuul job to use the "min" devstack version that is used
    by other repos such as 'fault' and avoid setting up the
    glance devstack plugin altogether.

    Change-Id: Id16671961e10962530d2eaff28387b4b206e0a3b
    Partial-Bug: 1840292
    Signed-off-by: Al Bailey <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/676753
Committed: https://git.openstack.org/cgit/starlingx/metal/commit/?id=5b9ae3a5ff8bda44d61230429e1aa4b4f56d3e4e
Submitter: Zuul
Branch: master

commit 5b9ae3a5ff8bda44d61230429e1aa4b4f56d3e4e
Author: Al Bailey <email address hidden>
Date: Thu Aug 15 13:50:14 2019 -0500

    Fixing failing devstack zuul job

    The glance devstack plugin is not working for us,
    and is not needed for our devstack to work, so updating
    the zuul job to use the 'min' devstack version that is used
    by other repos such as 'fault' and avoid setting up the
    glance devstack plugin altogether.

    Change-Id: Ibad252c69c19d425fe3c9f9b85e773286b3e3c43
    Partial-Bug: 1840292
    Signed-off-by: Al Bailey <email address hidden>