devstack

Devstack installs version-less rdo-release package

Bug #1833696 reported by Carlos Goncalves on 2019-06-21

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Shared File Systems Service (Manila)	Fix Released	High	Goutham Pacha Ravi	OpenStack Shared File Systems Service (Manila) train-3
	devstack	Fix Released	Undecided	Carlos Goncalves

Bug Description

Keystone database migration fails on stable/queens and CentOS 7.

Devstack installs package rdo-release from https://rdoproject.org/repos/rdo-release.rpm which redirects to latest stable, Stein at present, and where MariaDB 10.3.z is available and gets installed. The keystone database migration later on at deployment fails. It requires MariaDB <10.3.

[vagrant@localhost devstack]$ git rev-parse HEAD
e3100794d1615fc6b4216f54e2f7859a33d60b05

[vagrant@localhost devstack]$ rpm -q rdo-release
rdo-release-stein-2.noarch

vagrant@localhost devstack]$ rpm -qa | grep maria
mariadb-config-10.3.10-1.el7.0.0.rdo2.x86_64
mariadb-devel-10.3.10-1.el7.0.0.rdo2.x86_64
mariadb-10.3.10-1.el7.0.0.rdo2.x86_64
mariadb-server-10.3.10-1.el7.0.0.rdo2.x86_64
mariadb-common-10.3.10-1.el7.0.0.rdo2.x86_64
mariadb-errmsg-10.3.10-1.el7.0.0.rdo2.x86_64
mariadb-libs-10.3.10-1.el7.0.0.rdo2.x86_64

vagrant@localhost devstack]$ ./stack.sh
[...]
INFO migrate.versioning.api [-] done
INFO migrate.versioning.api [-] 32 -> 33...
INFO migrate.versioning.api [-] done
INFO migrate.versioning.api [-] 33 -> 34...
INFO migrate.versioning.api [-] done
INFO migrate.versioning.api [-] 34 -> 35...
ERROR oslo_db.sqlalchemy.exc_filters [-] DBAPIError exception wrapped from (pymysql.err.ProgrammingError) (1064, u"You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax
to use near 'VARCHAR(64)' at line 1") [SQL: u'\nALTER TABLE application_credential ADD system VARCHAR(64)'] (Background on this error at: http://sqlalche.me/e/f405): ProgrammingError: (1064, u"You have an error in your SQL syntax; check
the manual that corresponds to your MariaDB server version for the right syntax to use near 'VARCHAR(64)' at line 1")
ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
ERROR oslo_db.sqlalchemy.exc_filters context)
ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
ERROR oslo_db.sqlalchemy.exc_filters cursor.execute(statement, parameters)
ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 165, in execute
ERROR oslo_db.sqlalchemy.exc_filters result = self._query(query)
ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 321, in _query
ERROR oslo_db.sqlalchemy.exc_filters conn.query(q)
ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 860, in query
ERROR oslo_db.sqlalchemy.exc_filters self._affected_rows = self._read_query_result(unbuffered=unbuffered)
ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1061, in _read_query_result
ERROR oslo_db.sqlalchemy.exc_filters result.read()
ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1349, in read
ERROR oslo_db.sqlalchemy.exc_filters first_packet = self.connection._read_packet()
ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1018, in _read_packet
ERROR oslo_db.sqlalchemy.exc_filters packet.check_error()
ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 384, in check_error
ERROR oslo_db.sqlalchemy.exc_filters err.raise_mysql_exception(self._data)
ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
ERROR oslo_db.sqlalchemy.exc_filters raise errorclass(errno, errval)
ERROR oslo_db.sqlalchemy.exc_filters ProgrammingError: (1064, u"You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'VARCHAR(64)' at line 1")
ERROR oslo_db.sqlalchemy.exc_filters
CRITICAL keystone [-] Unhandled error: DBMigrationError: (pymysql.err.ProgrammingError) (1064, u"You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'VARCHAR(64)' at line 1") [SQL: u'\nALTER TABLE application_credential ADD system VARCHAR(64)'] (Background on this error at: http://sqlalche.me/e/f405)
ERROR keystone Traceback (most recent call last):
ERROR keystone File "/usr/bin/keystone-manage", line 10, in <module>
ERROR keystone sys.exit(main())
ERROR keystone File "/opt/stack/keystone/keystone/cmd/manage.py", line 45, in main
ERROR keystone cli.main(argv=sys.argv, config_files=config_files)
ERROR keystone File "/opt/stack/keystone/keystone/cmd/cli.py", line 1349, in main
ERROR keystone CONF.command.cmd_class.main()
ERROR keystone File "/opt/stack/keystone/keystone/cmd/cli.py", line 548, in main
ERROR keystone CONF.command.version)
ERROR keystone File "/opt/stack/keystone/keystone/common/sql/upgrades.py", line 250, in offline_sync_database_to_version
ERROR keystone expand_schema()
ERROR keystone File "/opt/stack/keystone/keystone/common/sql/upgrades.py", line 325, in expand_schema
ERROR keystone _sync_repo(repo_name=EXPAND_REPO)
ERROR keystone File "/opt/stack/keystone/keystone/common/sql/upgrades.py", line 187, in _sync_repo
ERROR keystone init_version=init_version, sanity_check=False)
ERROR keystone File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/migration.py", line 81, in db_sync
ERROR keystone raise exception.DBMigrationError(ex)
ERROR keystone DBMigrationError: (pymysql.err.ProgrammingError) (1064, u"You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'VARCHAR(64)' at line 1") [SQL: u'\nALTER TABLE application_credential ADD system VARCHAR(64)'] (Background on this error at: http://sqlalche.me/e/f405)
ERROR keystone
+lib/keystone:init_keystone:1 exit_trap
+./stack.sh:exit_trap:510 local r=1
++./stack.sh:exit_trap:511 jobs -p
+./stack.sh:exit_trap:511 jobs=
+./stack.sh:exit_trap:514 [[ -n '' ]]
+./stack.sh:exit_trap:520 '[' -f /tmp/tmp.pFdNvvsTTG ']'
+./stack.sh:exit_trap:521 rm /tmp/tmp.pFdNvvsTTG
+./stack.sh:exit_trap:525 kill_spinner
+./stack.sh:kill_spinner:424 '[' '!' -z '' ']'
+./stack.sh:exit_trap:527 [[ 1 -ne 0 ]]
+./stack.sh:exit_trap:528 echo 'Error on exit'
Error on exit
+./stack.sh:exit_trap:530 type -p generate-subunit
+./stack.sh:exit_trap:531 generate-subunit 1561105010 823 fail
+./stack.sh:exit_trap:533 [[ -z /opt/stack/logs ]]
+./stack.sh:exit_trap:536 /home/vagrant/devstack/tools/worlddump.py -d /opt/stack/logs
World dumping... see /opt/stack/logs/worlddump-2019-06-21-083033.txt for details
+./stack.sh:exit_trap:545 exit 1
[vagrant@localhost devstack]$

Tags:

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-06-21: Fix proposed to devstack (master)

Fix proposed to branch: master
Review: https://review.opendev.org/666817

Changed in devstack:
assignee:	nobody → Carlos Goncalves (cgoncalves)
status:	New → In Progress

Revision history for this message

Goutham Pacha Ravi (gouthamr) wrote on 2019-06-26:

Manila runs a CentOS job that's affected by this bug. Testing if https://review.opendev.org/666817 fixes the issue here: https://review.opendev.org/#/c/667721/

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-07-02: Fix merged to devstack (master)

Reviewed: https://review.opendev.org/666817
Committed: https://git.openstack.org/cgit/openstack/devstack/commit/?id=8b31dce38b630ba3ed1883bcf91a90d296ae44f2
Submitter: Zuul
Branch: master

commit 8b31dce38b630ba3ed1883bcf91a90d296ae44f2
Author: Carlos Goncalves <email address hidden>
Date: Fri Jun 21 13:13:40 2019 +0200

Fix rdo-release install

    The URL for rdo-release package is version-less and redirects to latest
    stable version. This becomes problematic when stacking older stable
    versions as dependencies might not be met or newer and incompatibile
    ones might get installed.

Closes-Bug: #1833696

Change-Id: Icb07dcb4c9a3950a3c31a3a8dcb8d0b4c713fdb1

Changed in devstack:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-07-02: Fix proposed to devstack (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/668720

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-07-03: Fix merged to devstack (stable/stein)

Reviewed: https://review.opendev.org/668720
Committed: https://git.openstack.org/cgit/openstack/devstack/commit/?id=fb3b6e4baa5a990da70c8b43bca181959a610278
Submitter: Zuul
Branch: stable/stein

commit fb3b6e4baa5a990da70c8b43bca181959a610278
Author: Carlos Goncalves <email address hidden>
Date: Fri Jun 21 13:13:40 2019 +0200

Fix rdo-release install

Closes-Bug: #1833696

Change-Id: Icb07dcb4c9a3950a3c31a3a8dcb8d0b4c713fdb1
(cherry picked from commit 8b31dce38b630ba3ed1883bcf91a90d296ae44f2)

tags:

added: in-stable-stein

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-07-03: Fix proposed to devstack (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/668787

Revision history for this message

Goutham Pacha Ravi (gouthamr) wrote on 2019-07-03:

Manila's stable/queens has a voting job that deploys openstack/manila with DevStack on CentOS [1]. So added manila as an affected project to this bug. Thanks for the fix Carlos! Hope we can backport to stable/queens soon.

Changed in manila:
importance:	Undecided → High
assignee:	nobody → Goutham Pacha Ravi (gouthamr)
milestone:	none → train-2

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-07-09: Fix merged to devstack (stable/rocky)

Reviewed: https://review.opendev.org/668787
Committed: https://git.openstack.org/cgit/openstack/devstack/commit/?id=ed120e335fadc901f9547e347e3c3a51b278bc73
Submitter: Zuul
Branch: stable/rocky

commit ed120e335fadc901f9547e347e3c3a51b278bc73
Author: Carlos Goncalves <email address hidden>
Date: Fri Jun 21 13:13:40 2019 +0200

Fix rdo-release install

Closes-Bug: #1833696

    Change-Id: Icb07dcb4c9a3950a3c31a3a8dcb8d0b4c713fdb1
    (cherry picked from commit 8b31dce38b630ba3ed1883bcf91a90d296ae44f2)
    (cherry picked from commit fb3b6e4baa5a990da70c8b43bca181959a610278)

tags:

added: in-stable-rocky

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-07-09: Fix proposed to devstack (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/669872

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-07-26: Fix merged to devstack (stable/queens)

#10

Reviewed: https://review.opendev.org/669872
Committed: https://git.openstack.org/cgit/openstack/devstack/commit/?id=6f3696c1680f59c60609d7037983e6cda5d5cdd3
Submitter: Zuul
Branch: stable/queens

commit 6f3696c1680f59c60609d7037983e6cda5d5cdd3
Author: Carlos Goncalves <email address hidden>
Date: Fri Jun 21 13:13:40 2019 +0200

Fix rdo-release install

Closes-Bug: #1833696

    Change-Id: Icb07dcb4c9a3950a3c31a3a8dcb8d0b4c713fdb1
    (cherry picked from commit 8b31dce38b630ba3ed1883bcf91a90d296ae44f2)
    (cherry picked from commit fb3b6e4baa5a990da70c8b43bca181959a610278)
    (cherry picked from commit ed120e335fadc901f9547e347e3c3a51b278bc73)

tags:

added: in-stable-queens

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-07-26: Fix proposed to manila (stable/queens)

#11

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/673011

Goutham Pacha Ravi (gouthamr) on 2019-07-26

Changed in manila:
status:	New → In Progress
milestone:	train-2 → train-3

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-07-26: Fix merged to manila (stable/queens)

#12

Reviewed: https://review.opendev.org/673011
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=9aa87e4cafc09a8452e2dbec6135b77a8e3f2b39
Submitter: Zuul
Branch: stable/queens

commit 9aa87e4cafc09a8452e2dbec6135b77a8e3f2b39
Author: Goutham Pacha Ravi <email address hidden>
Date: Fri Jul 26 08:59:11 2019 -0700

[stable/queens-only][ci] Enable LVM job

    The LVM job in the check queue was set to non-voting
    and the gate job was disabled due to bug 1833696 in
    DevStack. It's now been fixed [1]. So revert changes
    to this job.

[1] https://review.opendev.org/#/c/669872/
Closes-Bug: #1833696

Change-Id: I3ce5cbee3ab82804871d7363eea53f319d96fce1

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-09-02: Fix proposed to devstack (stable/ocata)

#13

Fix proposed to branch: stable/ocata
Review: https://review.opendev.org/679628