'nova-manage db online_data_migrations' count fail

[iain MacDonnell]

'nova-manage db online_data_migrations' attempts to display summary counts of migrations "Needed" and "Completed" in a pretty table at the end, but fails to accumulate the totals between successive invocations of _run_migration(), and ends up reporting zeroes.

# nova-manage db online_data_migrations
/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py:332: NotSupportedWarning: Configuration option(s) ['use_tpool'] not supported
  exception.NotSupportedWarning
Running batches of 50 until complete
/usr/lib/python2.7/site-packages/pymysql/cursors.py:166: Warning: (3090, u"Changing sql mode 'NO_AUTO_CREATE_USER' is deprecated. It will be removed in a future release.")
  result = self._query(query)
2 rows matched query migrate_instances_add_request_spec, 0 migrated
13 rows matched query migrate_quota_limits_to_api_db, 13 migrated
37 rows matched query populate_uuids, 37 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
50 rows matched query populate_uuids, 50 migrated
21 rows matched query populate_uuids, 21 migrated
+---------------------------------------------+--------------+-----------+
| Migration | Total Needed | Completed |
+---------------------------------------------+--------------+-----------+
| delete_build_requests_with_no_instance_uuid | 0 | 0 |
| migrate_aggregate_reset_autoincrement | 0 | 0 |
| migrate_aggregates | 0 | 0 |
| migrate_instance_groups_to_api_db | 0 | 0 |
| migrate_instances_add_request_spec | 0 | 0 |
| migrate_keypairs_to_api_db | 0 | 0 |
| migrate_quota_classes_to_api_db | 0 | 0 |
| migrate_quota_limits_to_api_db | 0 | 0 |
| migration_migrate_to_uuid | 0 | 0 |
| populate_uuids | 0 | 0 |
| service_uuids_online_data_migration | 0 | 0 |
+---------------------------------------------+--------------+-----------+

[Matt Riedemann (mriedem)]

(3:05:42 PM) mriedem: that populate_uuids migration was added in queens, and looking at a grenade run from queens, we should see bdms getting migrated to have a uuid, and i'm not seeing any results in the table for a grenade run on queens
(3:05:42 PM) mriedem: http://logs.openstack.org/48/604448/1/check/neutron-grenade/19d8166/logs/grenade.sh.txt.gz
(3:05:46 PM) mriedem: so yeah, likely busted

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/605329

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/605828

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/605836

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/605839

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/605840

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/605842

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/605329
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c4c6dc736e084f2d919b9bcf93d26df87e0341c9
Submitter: Zuul
Branch: master

commit c4c6dc736e084f2d919b9bcf93d26df87e0341c9
Author: imacdonn <email address hidden>
Date: Wed Sep 26 04:30:49 2018 +0000

    nova-manage - fix online_data_migrations counts

    When running online_data_migrations in batches, totals were not
    being accumulated - rather the counts each batch would clobber
    those from the previous one, and the last batch would run no
    migrations, so the totals were reported as zero.

    Change-Id: Ib616f2efb69baa16e18601d27b747220bbefeb16
    Closes-Bug: #1794364

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/605828
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=fc9c8772012b528176c3f58d99a58491bb1aedb4
Submitter: Zuul
Branch: stable/rocky

commit fc9c8772012b528176c3f58d99a58491bb1aedb4
Author: imacdonn <email address hidden>
Date: Wed Sep 26 04:30:49 2018 +0000

    nova-manage - fix online_data_migrations counts

    When running online_data_migrations in batches, totals were not
    being accumulated - rather the counts each batch would clobber
    those from the previous one, and the last batch would run no
    migrations, so the totals were reported as zero.

    Change-Id: Ib616f2efb69baa16e18601d27b747220bbefeb16
    Closes-Bug: #1794364
    (cherry picked from commit c4c6dc736e084f2d919b9bcf93d26df87e0341c9)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/605839
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d8b99f05abb2fa8964a326958804f47daf95da64
Submitter: Zuul
Branch: stable/queens

commit d8b99f05abb2fa8964a326958804f47daf95da64
Author: imacdonn <email address hidden>
Date: Wed Sep 26 04:30:49 2018 +0000

    nova-manage - fix online_data_migrations counts

    When running online_data_migrations in batches, totals were not
    being accumulated - rather the counts each batch would clobber
    those from the previous one, and the last batch would run no
    migrations, so the totals were reported as zero.

    Change-Id: Ib616f2efb69baa16e18601d27b747220bbefeb16
    Closes-Bug: #1794364
    (cherry picked from commit c4c6dc736e084f2d919b9bcf93d26df87e0341c9)
    (cherry picked from commit fc9c8772012b528176c3f58d99a58491bb1aedb4)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/605840
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=25e73d5cbf66984f69deb0e808a25f97772bd9dd
Submitter: Zuul
Branch: stable/pike

commit 25e73d5cbf66984f69deb0e808a25f97772bd9dd
Author: imacdonn <email address hidden>
Date: Wed Sep 26 04:30:49 2018 +0000

    nova-manage - fix online_data_migrations counts

    When running online_data_migrations in batches, totals were not
    being accumulated - rather the counts each batch would clobber
    those from the previous one, and the last batch would run no
    migrations, so the totals were reported as zero.

    Change-Id: Ib616f2efb69baa16e18601d27b747220bbefeb16
    Closes-Bug: #1794364
    (cherry picked from commit c4c6dc736e084f2d919b9bcf93d26df87e0341c9)
    (cherry picked from commit fc9c8772012b528176c3f58d99a58491bb1aedb4)
    (cherry picked from commit d8b99f05abb2fa8964a326958804f47daf95da64)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/605842
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5d33e554e4c96cec85274e8f46f3a7fa2b040b17
Submitter: Zuul
Branch: stable/ocata

commit 5d33e554e4c96cec85274e8f46f3a7fa2b040b17
Author: imacdonn <email address hidden>
Date: Wed Sep 26 04:30:49 2018 +0000

    nova-manage - fix online_data_migrations counts

    When running online_data_migrations in batches, totals were not
    being accumulated - rather the counts each batch would clobber
    those from the previous one, and the last batch would run no
    migrations, so the totals were reported as zero.

    Change-Id: Ib616f2efb69baa16e18601d27b747220bbefeb16
    Closes-Bug: #1794364
    (cherry picked from commit c4c6dc736e084f2d919b9bcf93d26df87e0341c9)
    (cherry picked from commit fc9c8772012b528176c3f58d99a58491bb1aedb4)
    (cherry picked from commit d8b99f05abb2fa8964a326958804f47daf95da64)
    (cherry picked from commit 25e73d5cbf66984f69deb0e808a25f97772bd9dd)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 18.0.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.1.6 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 15.1.5 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.7 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/611463

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/605836
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c86f309c564f4a394285068bbdb4f138cfff1651
Submitter: Zuul
Branch: master

commit c86f309c564f4a394285068bbdb4f138cfff1651
Author: Matt Riedemann <email address hidden>
Date: Thu Sep 27 14:53:10 2018 -0400

    Add more documentation for online_data_migrations CLI

    This is a follow up to commit c4c6dc736 to clarify some
    confusing comments in the code, add more comments in
    the actual runtime code, and also provide an example
    in the CLI man page docs along with an explanation of
    the output, specifically for the case that $found>0
    but done=0 and what that means.

    Change-Id: I0691caab2c44d3189504c54e51bb263ecdc5d1d2
    Related-Bug: #1794364

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/611463
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=d47486d317e8cdb1cdab50d73f6484289ab082d4
Submitter: Zuul
Branch: master

commit d47486d317e8cdb1cdab50d73f6484289ab082d4
Author: imacdonn <email address hidden>
Date: Wed Oct 17 22:48:18 2018 +0000

    cinder-manage online_data_migrations fixes

    Addresses some issues with this command:

    1) When used without the --max-count option, the summary table will
       always show zero migrations run, because it only accounts for the
       last batch, and the loop only exits when the last batch does no work.

    2) "remaining" counts cannot be accurate, given the way migrations are
       implemented, because the "found" count refers to the number of rows
       that exist in the database, not the number that still need the
       migration applied.

    3) In the case where no migrations are successful, but some raise
       exceptions, the command was exiting with status zero, which usually
       indicates "success". This can cause issues that cause migration
       failures to go unnoticed, especially when automated.

    4) When exceptions do occur, a minimally useful message is output, and
       no detail about the exception is available to the user. The exception
       detail should be logged.

    5) Inaccuracies in the documentation - "--max_number" should be
       "--max-count", and stale references to the "--ignore_state" option,
       which was removed in [1]

    The solution for (3) introduces a new exit status, 2. See release note
    for details.

    These changes are aligned with equivalents [2][3] for the nova-manage
    command, except for the calculation of "Total Needed" - nova seems to
    interpret the "found" count differently/inconsistently.

    [1] https://review.openstack.org/510201
    [2] https://review.openstack.org/605828
    [3] https://review.openstack.org/608091

    Change-Id: I878480eb2359625cde839b073230844acc645cba
    Closes-Bug: #1794364
    Closes-Bug: #1796192

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/614617

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/614617
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=74fd810ad1a8561ba150925d19feb8cbe598fe84
Submitter: Zuul
Branch: stable/rocky

commit 74fd810ad1a8561ba150925d19feb8cbe598fe84
Author: imacdonn <email address hidden>
Date: Wed Oct 17 22:48:18 2018 +0000

    cinder-manage online_data_migrations fixes

    Addresses some issues with this command:

    1) When used without the --max-count option, the summary table will
       always show zero migrations run, because it only accounts for the
       last batch, and the loop only exits when the last batch does no work.

    2) "remaining" counts cannot be accurate, given the way migrations are
       implemented, because the "found" count refers to the number of rows
       that exist in the database, not the number that still need the
       migration applied.

    3) In the case where no migrations are successful, but some raise
       exceptions, the command was exiting with status zero, which usually
       indicates "success". This can cause issues that cause migration
       failures to go unnoticed, especially when automated.

    4) When exceptions do occur, a minimally useful message is output, and
       no detail about the exception is available to the user. The exception
       detail should be logged.

    5) Inaccuracies in the documentation - "--max_number" should be
       "--max-count", and stale references to the "--ignore_state" option,
       which was removed in [1]

    The solution for (3) introduces a new exit status, 2. See release note
    for details.

    These changes are aligned with equivalents [2][3] for the nova-manage
    command, except for the calculation of "Total Needed" - nova seems to
    interpret the "found" count differently/inconsistently.

    [1] https://review.openstack.org/510201
    [2] https://review.openstack.org/605828
    [3] https://review.openstack.org/608091

    Change-Id: I878480eb2359625cde839b073230844acc645cba
    Closes-Bug: #1794364
    Closes-Bug: #1796192
    (cherry picked from commit d47486d317e8cdb1cdab50d73f6484289ab082d4)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 13.0.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 14.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.