nova-manage db archive_deleted_rows is not multi-cell aware

[Matt Riedemann]

With the superconductor mode change in devstack in pike, the post_test_hook in the nova-next CI job is only cleaning up entries from the cell0 database:

http://logs.openstack.org/58/506458/2/check/gate-tempest-dsvm-neutron-nova-next-full-ubuntu-xenial-nv/0d589b6/logs/devstack-gate-post_test_hook.txt.gz

This is because there are 3 config files now:

stack@devstack:~$ ls -l /etc/nova/
total 40
-rw-r--r-- 1 stack stack 3001 Sep 25 18:23 api-paste.ini
-rw-r--r-- 1 stack stack 395 Sep 25 18:23 nova-api-uwsgi.ini
-rw-r--r-- 1 stack stack 724 Sep 25 18:23 nova_cell1.conf
-rw-r--r-- 1 stack stack 3495 Sep 25 18:54 nova.conf
-rw-r--r-- 1 stack stack 3561 Sep 25 18:58 nova-cpu.conf
-rw-r--r-- 1 root root 2525 Sep 25 18:23 nova-dhcpbridge.conf
-rw-r--r-- 1 stack stack 349 Sep 25 18:23 nova-metadata-uwsgi.ini
-rw-r--r-- 1 stack stack 405 Sep 25 18:24 placement-uwsgi.ini
-rw-r--r-- 1 root root 941 Sep 25 18:23 rootwrap.conf
drwxr-xr-x 2 root root 4096 Sep 25 18:23 rootwrap.d

nova.conf is configured for the API database and cell0 so it only removes entries from cell0.

We could have nova-manage db archive_deleted_rows get the cells from the API database and iterate the cells to archive deleted rows.

This might require a --fanout or --all-cells option to the command though so users would opt into the behavior.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/550212

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/550212

[melanie witt (melwitt)]

Administrative: for some reason, the bug bot didn't add a link to the patch that was originally proposed for this launchpad:

https://review.openstack.org/507486

NOTE: the patch is old and in merge conflict at this point.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/672604

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/672840

[OpenStack Infra (hudson-openstack)]

Change abandoned by melanie witt (<email address hidden>) on branch: master
Review: https://review.opendev.org/672604
Reason: RIP, patch. It turns out that last context manager still has usefulness.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/507486
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=97b8cb3f5804cd39b7eb3ac2d481e43ceb52f2cf
Submitter: Zuul
Branch: master

commit 97b8cb3f5804cd39b7eb3ac2d481e43ceb52f2cf
Author: Kevin_Zheng <email address hidden>
Date: Tue Sep 26 19:02:00 2017 +0800

    nova-manage db archive_deleted_rows is not multi-cell aware

    The archive_deleted_rows cmd depend on DB connection config from config
    file, and when applying super-conductor mode, there are several config
    files for different cells. If so, the command can only archive rows in
    cell0 DB as it only reads the nova.conf

    This patch added code that provides --all-cells parameter to the
    command and read info for all cells from the api_db and then archive
    rows across all cells.

    The --all-cells parameter is passed on to the purge command when
    archive_deleted_rows is called with both --all-cells and --purge.

    Co-Authored-By: melanie witt <email address hidden>

    Change-Id: Id16c3d91d9ce5db9ffd125b59fffbfedf4a6843d
    Closes-Bug: #1719487

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/672840
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1c9de9c7779b1faf9d9542b3e5bd20da70067365
Submitter: Zuul
Branch: master

commit 1c9de9c7779b1faf9d9542b3e5bd20da70067365
Author: melanie witt <email address hidden>
Date: Fri Jul 26 00:06:18 2019 +0000

    Verify archive_deleted_rows --all-cells in post test hook

    We are already running archive_deleted_rows in the gate, but we are
    not verifying whether all instance records, for example, were actually
    successfully removed from the databases (cell0 and cell1).

    This adds the --all-cells option to our archive_deleted_rows runs and
    verifies that instance records were successfully removed from all cell
    databases.

    It is not sufficient to check only for return code 0 because
    archive_deleted_rows will still return 0 when it misses archiving
    records in cell databases.

    Related-Bug: #1719487

    Change-Id: If133b12bf02d708c099504a88b474dce0bdb0f00

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 20.0.0.0rc1 release candidate.