gnocchi-upgrade fails during Newton -> Ocata overcloud upgrade

[Marius Cornea]

gnocchi-upgrade fails during Newton -> Ocata overcloud upgrade:

2017-02-08 11:14:45.269 936196 INFO alembic.runtime.migration [-] Running upgrade aba5a217ca9b -> 397987e38570, Remove slashes from original resource IDs, recompute their id with creator
2017-02-08 11:14:45.333 936196 CRITICAL gnocchi [-] DBNonExistentConstraint: (pymysql.err.InternalError) (1091, u"Can't DROP 'fk_metric_resource_id_resource_id'; check that column/key exists") [SQL: u'ALTER TABLE metric DROP FOREIGN KEY fk_metric_resource_id_resource_id']

Attaching the gnocchi-upgrade.log.

[Pradeep Kilambi (pkilambi)]

looks like this line is failing during db migration:

https://github.com/openstack/gnocchi/blob/master/gnocchi/indexer/alembic/versions/397987e38570_no_more_slash_and_reencode.py#L90-L91

We'll need to check if the fk exists before dropping the constraint.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/430939

[Marius Cornea (mcornea)]

Quick update on this:

## Before running the upgrade, on fresh Newton installation:

## gnocchi before upgrade, newton
MariaDB [(none)]> use gnocchi;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [gnocchi]> SHOW CREATE TABLE metric;
+--------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+--------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| metric | CREATE TABLE `metric` (
  `id` binary(16) NOT NULL,
  `archive_policy_name` varchar(255) NOT NULL,
  `created_by_user_id` varchar(255) DEFAULT NULL,
  `created_by_project_id` varchar(255) DEFAULT NULL,
  `resource_id` binary(16) DEFAULT NULL,
  `name` varchar(255) DEFAULT NULL,
  `unit` va...

[Pradeep Kilambi (pkilambi)]

Could you try this patch and see if it helps: https://review.openstack.org/#/c/431101/

that migration is broken after the drop line as indicated in this patch.

[gordon chung (chungg)]

did you run upgrade, fail and try again? there is only one migration between 3.0 and 3.1 that involves dropping fk_metric_resource_id_resource_id constraint and you seem to be failing on it.

[Marius Cornea (mcornea)]

https://review.openstack.org/#/c/431101/ didn't help, I get the same missing fk_metric_resource_id_resource_id error.

Yes, I tried to manually run gnocchi-upgrade after the first failure which occurred during the overcloud upgrade process.

[gordon chung (chungg)]

https://review.openstack.org/#/c/431101/ shouldn't help unless you ran upgrade, failed, and tried again. right now it really seems like you are getting this error because you are running upgrade on after a previous failure.

[Marius Cornea (mcornea)]

It moved forward after applying https://review.openstack.org/#/c/431101/ but now it fails with:
ERROR gnocchi DBError: (pymysql.err.InternalError) (1138, u'Invalid use of NULL value') [SQL: u'ALTER TABLE resource MODIFY original_resource_id VARCHAR(255) NOT NULL']

full trace: http://paste.openstack.org/show/598442/

[OpenStack Infra (hudson-openstack)]

Change abandoned by Pradeep Kilambi (<email address hidden>) on branch: master
Review: https://review.openstack.org/430939

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/432431

[Marius Cornea (mcornea)]

gnocchi-upgrade finished successfuly after applying https://review.openstack.org/#/c/431101/ and https://review.openstack.org/#/c/432431/

[gordon chung (chungg)]

dope. thanks for confirming.

i'll need to wait until next week to get either Mehdi to merge items. everyone is on holidays currently.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/432431
Committed: https://git.openstack.org/cgit/openstack/gnocchi/commit/?id=9b53bce0c5fb86fc8db3055d861d7b1f2715a00c
Submitter: Jenkins
Branch: master

commit 9b53bce0c5fb86fc8db3055d861d7b1f2715a00c
Author: gord chung <email address hidden>
Date: Fri Feb 10 14:22:04 2017 -0500

    ensure original_resource_id is not none

    statsd resource doesn't have an original_resource_id so we need to
    set it by copying over id during upgrade

    Change-Id: Id516c5b1d091ca01688d6ad8cf1076a998d15333
    Closes-Bug: #1662849

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/3.1
Review: https://review.openstack.org/433431

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/433431
Committed: https://git.openstack.org/cgit/openstack/gnocchi/commit/?id=04ef146e1961448a1f0ad13075f6bc7fb0e19477
Submitter: Jenkins
Branch: stable/3.1

commit 04ef146e1961448a1f0ad13075f6bc7fb0e19477
Author: gord chung <email address hidden>
Date: Fri Feb 10 14:22:04 2017 -0500

    ensure original_resource_id is not none

    statsd resource doesn't have an original_resource_id so we need to
    set it by copying over id during upgrade

    Change-Id: Id516c5b1d091ca01688d6ad8cf1076a998d15333
    Closes-Bug: #1662849
    (cherry picked from commit 9b53bce0c5fb86fc8db3055d861d7b1f2715a00c)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/gnocchi 3.1.1 release.