'nova-manage db online_data_migrations' count fail

Bug #1794364 reported by iain MacDonnell on 2018-09-25
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Undecided
iain MacDonnell
OpenStack Compute (nova)
High
iain MacDonnell
Ocata
High
Matt Riedemann
Pike
High
Matt Riedemann
Queens
High
Matt Riedemann
Rocky
High
Matt Riedemann

Bug Description

'nova-manage db online_data_migrations' attempts to display summary counts of migrations "Needed" and "Completed" in a pretty table at the end, but fails to accumulate the totals between successive invocations of _run_migration(), and ends up reporting zeroes.

# nova-manage db online_data_migrations
/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py:332: NotSupportedWarning: Configuration option(s) ['use_tpool'] not supported
  exception.NotSupportedWarning
Running batches of 50 until complete
/usr/lib/python2.7/site-packages/pymysql/cursors.py:166: Warning: (3090, u"Changing sql mode 'NO_AUTO_CREATE_USER' is deprecated. It will be removed in a future release.")
  result = self._query(query)
2 rows matched query migrate_instances_add_request_spec, 0 migrated
13 rows matched query migrate_quota_limits_to_api_db, 13 migrated
37 rows matched query populate_uuids, 37 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
21 rows matched query populate_uuids, 21 migrated
+---------------------------------------------+--------------+-----------+
| Migration | Total Needed | Completed |
+---------------------------------------------+--------------+-----------+
| delete_build_requests_with_no_instance_uuid | 0 | 0 |
| migrate_aggregate_reset_autoincrement | 0 | 0 |
| migrate_aggregates | 0 | 0 |
| migrate_instance_groups_to_api_db | 0 | 0 |
| migrate_instances_add_request_spec | 0 | 0 |
| migrate_keypairs_to_api_db | 0 | 0 |
| migrate_quota_classes_to_api_db | 0 | 0 |
| migrate_quota_limits_to_api_db | 0 | 0 |
| migration_migrate_to_uuid | 0 | 0 |
| populate_uuids | 0 | 0 |
| service_uuids_online_data_migration | 0 | 0 |
+---------------------------------------------+--------------+-----------+

melanie witt (melwitt) on 2018-09-25
tags: added: nova-manage
Matt Riedemann (mriedem) wrote :

(3:05:42 PM) mriedem: that populate_uuids migration was added in queens, and looking at a grenade run from queens, we should see bdms getting migrated to have a uuid, and i'm not seeing any results in the table for a grenade run on queens
(3:05:42 PM) mriedem: http://logs.openstack.org/48/604448/1/check/neutron-grenade/19d8166/logs/grenade.sh.txt.gz
(3:05:46 PM) mriedem: so yeah, likely busted

Changed in nova:
status: New → Confirmed
importance: Undecided → High
tags: added: upgrade
Changed in nova:
assignee: nobody → iain MacDonnell (imacdonn)
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/605329
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c4c6dc736e084f2d919b9bcf93d26df87e0341c9
Submitter: Zuul
Branch: master

commit c4c6dc736e084f2d919b9bcf93d26df87e0341c9
Author: imacdonn <email address hidden>
Date: Wed Sep 26 04:30:49 2018 +0000

    nova-manage - fix online_data_migrations counts

    When running online_data_migrations in batches, totals were not
    being accumulated - rather the counts each batch would clobber
    those from the previous one, and the last batch would run no
    migrations, so the totals were reported as zero.

    Change-Id: Ib616f2efb69baa16e18601d27b747220bbefeb16
    Closes-Bug: #1794364

Changed in nova:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/605828
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=fc9c8772012b528176c3f58d99a58491bb1aedb4
Submitter: Zuul
Branch: stable/rocky

commit fc9c8772012b528176c3f58d99a58491bb1aedb4
Author: imacdonn <email address hidden>
Date: Wed Sep 26 04:30:49 2018 +0000

    nova-manage - fix online_data_migrations counts

    When running online_data_migrations in batches, totals were not
    being accumulated - rather the counts each batch would clobber
    those from the previous one, and the last batch would run no
    migrations, so the totals were reported as zero.

    Change-Id: Ib616f2efb69baa16e18601d27b747220bbefeb16
    Closes-Bug: #1794364
    (cherry picked from commit c4c6dc736e084f2d919b9bcf93d26df87e0341c9)

Reviewed: https://review.openstack.org/605839
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d8b99f05abb2fa8964a326958804f47daf95da64
Submitter: Zuul
Branch: stable/queens

commit d8b99f05abb2fa8964a326958804f47daf95da64
Author: imacdonn <email address hidden>
Date: Wed Sep 26 04:30:49 2018 +0000

    nova-manage - fix online_data_migrations counts

    When running online_data_migrations in batches, totals were not
    being accumulated - rather the counts each batch would clobber
    those from the previous one, and the last batch would run no
    migrations, so the totals were reported as zero.

    Change-Id: Ib616f2efb69baa16e18601d27b747220bbefeb16
    Closes-Bug: #1794364
    (cherry picked from commit c4c6dc736e084f2d919b9bcf93d26df87e0341c9)
    (cherry picked from commit fc9c8772012b528176c3f58d99a58491bb1aedb4)

Reviewed: https://review.openstack.org/605840
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=25e73d5cbf66984f69deb0e808a25f97772bd9dd
Submitter: Zuul
Branch: stable/pike

commit 25e73d5cbf66984f69deb0e808a25f97772bd9dd
Author: imacdonn <email address hidden>
Date: Wed Sep 26 04:30:49 2018 +0000

    nova-manage - fix online_data_migrations counts

    When running online_data_migrations in batches, totals were not
    being accumulated - rather the counts each batch would clobber
    those from the previous one, and the last batch would run no
    migrations, so the totals were reported as zero.

    Change-Id: Ib616f2efb69baa16e18601d27b747220bbefeb16
    Closes-Bug: #1794364
    (cherry picked from commit c4c6dc736e084f2d919b9bcf93d26df87e0341c9)
    (cherry picked from commit fc9c8772012b528176c3f58d99a58491bb1aedb4)
    (cherry picked from commit d8b99f05abb2fa8964a326958804f47daf95da64)

Reviewed: https://review.openstack.org/605842
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5d33e554e4c96cec85274e8f46f3a7fa2b040b17
Submitter: Zuul
Branch: stable/ocata

commit 5d33e554e4c96cec85274e8f46f3a7fa2b040b17
Author: imacdonn <email address hidden>
Date: Wed Sep 26 04:30:49 2018 +0000

    nova-manage - fix online_data_migrations counts

    When running online_data_migrations in batches, totals were not
    being accumulated - rather the counts each batch would clobber
    those from the previous one, and the last batch would run no
    migrations, so the totals were reported as zero.

    Change-Id: Ib616f2efb69baa16e18601d27b747220bbefeb16
    Closes-Bug: #1794364
    (cherry picked from commit c4c6dc736e084f2d919b9bcf93d26df87e0341c9)
    (cherry picked from commit fc9c8772012b528176c3f58d99a58491bb1aedb4)
    (cherry picked from commit d8b99f05abb2fa8964a326958804f47daf95da64)
    (cherry picked from commit 25e73d5cbf66984f69deb0e808a25f97772bd9dd)

This issue was fixed in the openstack/nova 18.0.2 release.

This issue was fixed in the openstack/nova 16.1.6 release.

This issue was fixed in the openstack/nova 15.1.5 release.

This issue was fixed in the openstack/nova 17.0.7 release.

Changed in cinder:
assignee: nobody → iain MacDonnell (imacdonn)

Fix proposed to branch: master
Review: https://review.openstack.org/611463

Changed in cinder:
status: New → In Progress

Reviewed: https://review.openstack.org/605836
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c86f309c564f4a394285068bbdb4f138cfff1651
Submitter: Zuul
Branch: master

commit c86f309c564f4a394285068bbdb4f138cfff1651
Author: Matt Riedemann <email address hidden>
Date: Thu Sep 27 14:53:10 2018 -0400

    Add more documentation for online_data_migrations CLI

    This is a follow up to commit c4c6dc736 to clarify some
    confusing comments in the code, add more comments in
    the actual runtime code, and also provide an example
    in the CLI man page docs along with an explanation of
    the output, specifically for the case that $found>0
    but done=0 and what that means.

    Change-Id: I0691caab2c44d3189504c54e51bb263ecdc5d1d2
    Related-Bug: #1794364

Reviewed: https://review.openstack.org/611463
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=d47486d317e8cdb1cdab50d73f6484289ab082d4
Submitter: Zuul
Branch: master

commit d47486d317e8cdb1cdab50d73f6484289ab082d4
Author: imacdonn <email address hidden>
Date: Wed Oct 17 22:48:18 2018 +0000

    cinder-manage online_data_migrations fixes

    Addresses some issues with this command:

    1) When used without the --max-count option, the summary table will
       always show zero migrations run, because it only accounts for the
       last batch, and the loop only exits when the last batch does no work.

    2) "remaining" counts cannot be accurate, given the way migrations are
       implemented, because the "found" count refers to the number of rows
       that exist in the database, not the number that still need the
       migration applied.

    3) In the case where no migrations are successful, but some raise
       exceptions, the command was exiting with status zero, which usually
       indicates "success". This can cause issues that cause migration
       failures to go unnoticed, especially when automated.

    4) When exceptions do occur, a minimally useful message is output, and
       no detail about the exception is available to the user. The exception
       detail should be logged.

    5) Inaccuracies in the documentation - "--max_number" should be
       "--max-count", and stale references to the "--ignore_state" option,
       which was removed in [1]

    The solution for (3) introduces a new exit status, 2. See release note
    for details.

    These changes are aligned with equivalents [2][3] for the nova-manage
    command, except for the calculation of "Total Needed" - nova seems to
    interpret the "found" count differently/inconsistently.

    [1] https://review.openstack.org/510201
    [2] https://review.openstack.org/605828
    [3] https://review.openstack.org/608091

    Change-Id: I878480eb2359625cde839b073230844acc645cba
    Closes-Bug: #1794364
    Closes-Bug: #1796192

Changed in cinder:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/614617
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=74fd810ad1a8561ba150925d19feb8cbe598fe84
Submitter: Zuul
Branch: stable/rocky

commit 74fd810ad1a8561ba150925d19feb8cbe598fe84
Author: imacdonn <email address hidden>
Date: Wed Oct 17 22:48:18 2018 +0000

    cinder-manage online_data_migrations fixes

    Addresses some issues with this command:

    1) When used without the --max-count option, the summary table will
       always show zero migrations run, because it only accounts for the
       last batch, and the loop only exits when the last batch does no work.

    2) "remaining" counts cannot be accurate, given the way migrations are
       implemented, because the "found" count refers to the number of rows
       that exist in the database, not the number that still need the
       migration applied.

    3) In the case where no migrations are successful, but some raise
       exceptions, the command was exiting with status zero, which usually
       indicates "success". This can cause issues that cause migration
       failures to go unnoticed, especially when automated.

    4) When exceptions do occur, a minimally useful message is output, and
       no detail about the exception is available to the user. The exception
       detail should be logged.

    5) Inaccuracies in the documentation - "--max_number" should be
       "--max-count", and stale references to the "--ignore_state" option,
       which was removed in [1]

    The solution for (3) introduces a new exit status, 2. See release note
    for details.

    These changes are aligned with equivalents [2][3] for the nova-manage
    command, except for the calculation of "Total Needed" - nova seems to
    interpret the "found" count differently/inconsistently.

    [1] https://review.openstack.org/510201
    [2] https://review.openstack.org/605828
    [3] https://review.openstack.org/608091

    Change-Id: I878480eb2359625cde839b073230844acc645cba
    Closes-Bug: #1794364
    Closes-Bug: #1796192
    (cherry picked from commit d47486d317e8cdb1cdab50d73f6484289ab082d4)

This issue was fixed in the openstack/cinder 13.0.2 release.

This issue was fixed in the openstack/cinder 14.0.0.0rc1 release candidate.

This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers