nova-status upgrade check shows warnings when it shouldn't
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| | OpenStack Compute (nova) |
High
|
Matt Riedemann | ||
| | Ocata |
High
|
Matt Riedemann | ||
| | Pike |
High
|
Matt Riedemann | ||
| | Queens |
High
|
Matt Riedemann | ||
| | Rocky |
High
|
Matt Riedemann | ||
Bug Description
# nova-status upgrade check
+------
| Upgrade Check Results |
+------
| Check: Cells v2 |
| Result: Success |
| Details: None |
+------
| Check: Placement API |
| Result: Success |
| Details: None |
+------
| Check: Resource Providers |
| Result: Warning |
| Details: There are no compute resource providers in the Placement |
| service but there are 2 compute nodes in the deployment. |
| This means no compute nodes are reporting into the |
| Placement service and need to be upgraded and/or fixed. |
| See |
| https:/
| for more details. |
+------
| Check: Ironic Flavor Migration |
| Result: Success |
| Details: None |
+------
| Check: API Service Version |
| Result: Success |
| Details: None |
+------
| Check: Request Spec Migration |
| Result: Success |
| Details: None |
+------
nova hypervisor-list
+------
| ID | Hypervisor hostname | State | Status |
+------
| UUID1 | node03.NOPE | up | enabled |
| UUID2 | node02.NOPE | up | enabled |
+------
openstack resource provider list
+------
| uuid | name | generation |
+------
| UUID1 | node02.NOPE | 76 |
| UUID2 | node03.NOPE | 34 |
+------
| Matt Riedemann (mriedem) wrote : | #1 |
| tags: | added: nova-status upgrade |
| Changed in nova: | |
| status: | New → Confirmed |
| importance: | Undecided → High |
| Matt Riedemann (mriedem) wrote : | #2 |
Here is a devstack patch that runs nova-status upgrade check which should show this same warning and actually make devstack fail:
| Matt Riedemann (mriedem) wrote : | #3 |
The devstack change didn't fail because when nova-status upgrade check runs it's not seeing any compute nodes in the cell databases:
2018-09-04 22:05:45.875 | +------
2018-09-04 22:05:45.875 | | Check: Resource Providers |
2018-09-04 22:05:45.875 | | Result: Success |
2018-09-04 22:05:45.875 | | Details: There are no compute resource providers in the Placement |
2018-09-04 22:05:45.875 | | service nor are there compute nodes in the database. |
2018-09-04 22:05:45.875 | | Remember to configure new compute nodes to report into the |
2018-09-04 22:05:45.876 | | Placement service. See |
2018-09-04 22:05:45.876 | | https:/
2018-09-04 22:05:45.876 | | for more details. |
2018-09-04 22:05:45.876 | +------
| Matt Riedemann (mriedem) wrote : | #4 |
Ah I think I know what's happening, the else block on this for loop is busted:
And that overwrites the num_computes value to 0 because there are no compute nodes in the cell0 databsae.
Fix proposed to branch: master
Review: https:/
| Changed in nova: | |
| assignee: | nobody → Matt Riedemann (mriedem) |
| status: | Confirmed → In Progress |
| tags: | added: placement |
Fix proposed to branch: stable/rocky
Review: https:/
Fix proposed to branch: stable/queens
Review: https:/
Fix proposed to branch: stable/pike
Review: https:/
Fix proposed to branch: stable/ocata
Review: https:/
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: master
commit dcd421ae9e6f039
Author: Matt Riedemann <email address hidden>
Date: Tue Sep 4 20:57:33 2018 -0400
Fix nova-status "_check_
The way in which this check counted compute nodes was
broken because of an incorrect for/else condition. If
the check is run with a nova.conf like we have in
devstack, where the API database is configured but
the [database]
there are no compute nodes, the check passes saying
there are no compute nodes even if the are compute
nodes found in the cell databases (in the for loop).
This is because the else executes because the for loop
doesn't break, and then _count_
for cell0 and overwrites the num_computes variable.
This fixes the issue by checking if we have cell mappings
before running the loop, else we hit the else block as
was originally intended.
Change-Id: I1a706d028a9ca8
Partial-Bug: #1790721
Fix proposed to branch: stable/rocky
Review: https:/
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: master
commit 63c10d2d536c3ff
Author: Matt Riedemann <email address hidden>
Date: Tue Sep 4 14:53:25 2018 -0400
Configure placement DB context manager for nova-manage/status
The create_
Rocky via change Id609789ef6b4a4
relies on hitting some tables in the API (or placement) database.
The DB API code for that migration relies on a placement context
manager which looks like it was regressed with change
I2fff528060
results in a DB error trying to query the projects table but
because of a generic try/except in _run_migration, the failure
was missed in CI testing.
Similarly, the nova-status upgrade check "_check_
routine also uses the placement DB API context manager to count the
number of compute resource providers in the API (or placement) DB,
which is returning 0 because it's not using the proper DB connection.
This was not caught in the nova-status CLI tests because they use
the DatabaseFixture which *does* configure the global placement DB
API context manager.
This adds the configuration of the global placement DB API context
manager so we can properly query the placement-related tables.
The blanket problematic try/except from _run_migration is left
as-is in this change but will be addressed in a separate patch.
Integration testing of this fix is being performed with devstack:
https:/
Change-Id: I9d97b7a904e2b7
Closes-Bug: #1790701
Closes-Bug: #1790721
| Changed in nova: | |
| status: | In Progress → Fix Released |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/rocky
commit f588a0b6c337368
Author: Matt Riedemann <email address hidden>
Date: Tue Sep 4 20:57:33 2018 -0400
Fix nova-status "_check_
The way in which this check counted compute nodes was
broken because of an incorrect for/else condition. If
the check is run with a nova.conf like we have in
devstack, where the API database is configured but
the [database]
there are no compute nodes, the check passes saying
there are no compute nodes even if the are compute
nodes found in the cell databases (in the for loop).
This is because the else executes because the for loop
doesn't break, and then _count_
for cell0 and overwrites the num_computes variable.
This fixes the issue by checking if we have cell mappings
before running the loop, else we hit the else block as
was originally intended.
Change-Id: I1a706d028a9ca8
Partial-Bug: #1790721
(cherry picked from commit dcd421ae9e6f039
| tags: | added: in-stable-rocky |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/rocky
commit 7f25c3e6072c2f8
Author: Matt Riedemann <email address hidden>
Date: Tue Sep 4 14:53:25 2018 -0400
Configure placement DB context manager for nova-manage/status
The create_
Rocky via change Id609789ef6b4a4
relies on hitting some tables in the API (or placement) database.
The DB API code for that migration relies on a placement context
manager which looks like it was regressed with change
I2fff528060
results in a DB error trying to query the projects table but
because of a generic try/except in _run_migration, the failure
was missed in CI testing.
Similarly, the nova-status upgrade check "_check_
routine also uses the placement DB API context manager to count the
number of compute resource providers in the API (or placement) DB,
which is returning 0 because it's not using the proper DB connection.
This was not caught in the nova-status CLI tests because they use
the DatabaseFixture which *does* configure the global placement DB
API context manager.
This adds the configuration of the global placement DB API context
manager so we can properly query the placement-related tables.
The blanket problematic try/except from _run_migration is left
as-is in this change but will be addressed in a separate patch.
Integration testing of this fix is being performed with devstack:
https:/
Change-Id: I9d97b7a904e2b7
Closes-Bug: #1790701
Closes-Bug: #1790721
(cherry picked from commit 63c10d2d536c3ff
This issue was fixed in the openstack/nova 18.0.1 release.
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/queens
commit 6a806de71b94550
Author: Matt Riedemann <email address hidden>
Date: Tue Sep 4 20:57:33 2018 -0400
Fix nova-status "_check_
The way in which this check counted compute nodes was
broken because of an incorrect for/else condition. If
the check is run with a nova.conf like we have in
devstack, where the API database is configured but
the [database]
there are no compute nodes, the check passes saying
there are no compute nodes even if the are compute
nodes found in the cell databases (in the for loop).
This is because the else executes because the for loop
doesn't break, and then _count_
for cell0 and overwrites the num_computes variable.
This fixes the issue by checking if we have cell mappings
before running the loop, else we hit the else block as
was originally intended.
Change-Id: I1a706d028a9ca8
Partial-Bug: #1790721
(cherry picked from commit dcd421ae9e6f039
(cherry picked from commit f588a0b6c337368
| tags: | added: in-stable-queens |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/pike
commit 3f562226112b99d
Author: Matt Riedemann <email address hidden>
Date: Tue Sep 4 20:57:33 2018 -0400
Fix nova-status "_check_
The way in which this check counted compute nodes was
broken because of an incorrect for/else condition. If
the check is run with a nova.conf like we have in
devstack, where the API database is configured but
the [database]
there are no compute nodes, the check passes saying
there are no compute nodes even if the are compute
nodes found in the cell databases (in the for loop).
This is because the else executes because the for loop
doesn't break, and then _count_
for cell0 and overwrites the num_computes variable.
This fixes the issue by checking if we have cell mappings
before running the loop, else we hit the else block as
was originally intended.
Change-Id: I1a706d028a9ca8
Partial-Bug: #1790721
(cherry picked from commit dcd421ae9e6f039
(cherry picked from commit f588a0b6c337368
(cherry picked from commit 6a806de71b94550
| tags: | added: in-stable-pike |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/ocata
commit ff8cb33283bbce4
Author: Matt Riedemann <email address hidden>
Date: Tue Sep 4 20:57:33 2018 -0400
Fix nova-status "_check_
The way in which this check counted compute nodes was
broken because of an incorrect for/else condition. If
the check is run with a nova.conf like we have in
devstack, where the API database is configured but
the [database]
there are no compute nodes, the check passes saying
there are no compute nodes even if the are compute
nodes found in the cell databases (in the for loop).
This is because the else executes because the for loop
doesn't break, and then _count_
for cell0 and overwrites the num_computes variable.
This fixes the issue by checking if we have cell mappings
before running the loop, else we hit the else block as
was originally intended.
Conflicts:
NOTE(mriedem): The conflict is due to not having change
I35206e665f
Change-Id: I1a706d028a9ca8
Partial-Bug: #1790721
(cherry picked from commit dcd421ae9e6f039
(cherry picked from commit f588a0b6c337368
(cherry picked from commit 6a806de71b94550
(cherry picked from commit 3f562226112b99d
| tags: | added: in-stable-ocata |
This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.
I suspect this is a similar regression as bug 1790701 in that the count for resource providers is using the placement DB API context manager from https:/