PCI device migration cannot continue with old deleted service records

Bug #1668310 reported by Dan Smith on 2017-02-27
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
High
Dan Smith
Mitaka
High
Matt Riedemann
Newton
High
Matt Riedemann
Ocata
High
Matt Riedemann

Bug Description

If deleted service records are present in the database, the Service minimum version calculation should ignore them, but it does not. One manifestation of this is the PCI device migration from mitaka/newton will never complete, emitting an error message like this:

2017-02-27 07:40:19.665 ERROR nova.db.sqlalchemy.api [req-ad21480f-613a-445b-a913-c54532b64ffa None None] Data migrations for PciDevice are not safe, likely because not all services that access the DB directly are updated to the latest version

Fix proposed to branch: master
Review: https://review.openstack.org/438578

Changed in nova:
assignee: nobody → Dan Smith (danms)
status: New → In Progress
Matt Riedemann (mriedem) on 2017-02-27
Changed in nova:
importance: Undecided → High
tags: added: db upgrades
Saverio Proto (zioproto) wrote :

I was able to workaround the problem like this:

delete from services where deleted != 0 ;
delete from aggregate_hosts where deleted !=0;
delete from aggregate_metadata where deleted !=0;
delete from aggregates where deleted !=0;

and then I also need nova to be at least at commit edbd3135f8087ea73dc73b06962bf8b869ee6f43

Reviewed: https://review.openstack.org/438578
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c79770e615799cd4457ac603dcad4fb3452fe2bc
Submitter: Jenkins
Branch: master

commit c79770e615799cd4457ac603dcad4fb3452fe2bc
Author: Dan Smith <email address hidden>
Date: Mon Feb 27 07:52:29 2017 -0800

    Ignore deleted services in minimum version calculation

    When we go to detect the minimum version for a given service, we
    should ignore any deleted services. Without this, we will return
    the minimum version of all records, including those that have been
    deleted with "nova service-delete". This patch filters deleted
    services from the query.

    Closes-Bug: #1668310
    Change-Id: Ic96a5eb3728f97a3c35d2c5121e6fdcd4fd1c70b

Changed in nova:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/438630
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1ad5c7305c37079ced24bf623810e63d5eac2661
Submitter: Jenkins
Branch: stable/newton

commit 1ad5c7305c37079ced24bf623810e63d5eac2661
Author: Dan Smith <email address hidden>
Date: Mon Feb 27 07:52:29 2017 -0800

    Ignore deleted services in minimum version calculation

    When we go to detect the minimum version for a given service, we
    should ignore any deleted services. Without this, we will return
    the minimum version of all records, including those that have been
    deleted with "nova service-delete". This patch filters deleted
    services from the query.

    Closes-Bug: #1668310
    Change-Id: Ic96a5eb3728f97a3c35d2c5121e6fdcd4fd1c70b
    (cherry picked from commit c79770e615799cd4457ac603dcad4fb3452fe2bc)
    (cherry picked from commit a1dd547d3ba7d9cf5b73da1ce9668e412501ace1)

Reviewed: https://review.openstack.org/438632
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2e05c82b807624b8420df4c94d68b75563b230d2
Submitter: Jenkins
Branch: stable/mitaka

commit 2e05c82b807624b8420df4c94d68b75563b230d2
Author: Dan Smith <email address hidden>
Date: Mon Feb 27 07:52:29 2017 -0800

    Ignore deleted services in minimum version calculation

    When we go to detect the minimum version for a given service, we
    should ignore any deleted services. Without this, we will return
    the minimum version of all records, including those that have been
    deleted with "nova service-delete". This patch filters deleted
    services from the query.

    Conflicts:
            nova/db/sqlalchemy/api.py
            nova/tests/unit/db/test_db_api.py

    NOTE(mriedem): The conflicts are due to not having change
    11cb56a2243faa9f2614a8f9a9a84603bc91d6b1 in Mitaka.

    Closes-Bug: #1668310
    Change-Id: Ic96a5eb3728f97a3c35d2c5121e6fdcd4fd1c70b
    (cherry picked from commit c79770e615799cd4457ac603dcad4fb3452fe2bc)
    (cherry picked from commit a1dd547d3ba7d9cf5b73da1ce9668e412501ace1)
    (cherry picked from commit 1ad5c7305c37079ced24bf623810e63d5eac2661)

Reviewed: https://review.openstack.org/438628
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a1dd547d3ba7d9cf5b73da1ce9668e412501ace1
Submitter: Jenkins
Branch: stable/ocata

commit a1dd547d3ba7d9cf5b73da1ce9668e412501ace1
Author: Dan Smith <email address hidden>
Date: Mon Feb 27 07:52:29 2017 -0800

    Ignore deleted services in minimum version calculation

    When we go to detect the minimum version for a given service, we
    should ignore any deleted services. Without this, we will return
    the minimum version of all records, including those that have been
    deleted with "nova service-delete". This patch filters deleted
    services from the query.

    Closes-Bug: #1668310
    Change-Id: Ic96a5eb3728f97a3c35d2c5121e6fdcd4fd1c70b
    (cherry picked from commit c79770e615799cd4457ac603dcad4fb3452fe2bc)

This issue was fixed in the openstack/nova 13.1.3 release.

This issue was fixed in the openstack/nova 15.0.1 release.

This issue was fixed in the openstack/nova 14.0.5 release.

This issue was fixed in the openstack/nova 16.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers