openstack server list --deleted --limit -1 hangs
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| | OpenStack Compute (nova) |
High
|
Matt Riedemann | ||
| | Ocata |
High
|
Matt Riedemann | ||
| | Pike |
High
|
Matt Riedemann | ||
| | Queens |
High
|
Matt Riedemann | ||
| | Rocky |
High
|
Matt Riedemann | ||
| | Stein |
High
|
Matt Riedemann | ||
| | Train |
High
|
Matt Riedemann | ||
Bug Description
OpenStack Rocky:
When running `openstack server list --deleted --limit -1 hangs` it will hang and not return.
A debug output is found in this pastebin
http://
and a direct curl is shown here
http://
It seems to be related to the marker as discussed here.
after the initial collection of 18 servers, for some reason it than tried to grab with
where the marker is the 18th deleted server in the list. it then loops on this until you breakout.
| Matt Riedemann (mriedem) wrote : | #1 |
| Changed in nova: | |
| assignee: | nobody → Matt Riedemann (mriedem) |
| status: | New → Triaged |
| importance: | Undecided → High |
| tags: | added: api |
| Changed in nova: | |
| status: | Triaged → In Progress |
| Matt Riedemann (mriedem) wrote : | #2 |
The regression was introduced with https:/
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: master
commit 45c2752f2ce08b3
Author: Matt Riedemann <email address hidden>
Date: Tue Oct 22 17:53:20 2019 -0400
Add functional regression test for bug 1849409
Change I1aa3ca6cc70cef
which was backported through to Newton, introduced a regression
when listing deleted servers with a marker because it assumes
that if BuildRequestLis
MarkerNotFound that the marker was found among the build requests
and does not account for that get_by_filters method short-circuiting
if filtering servers with deleted/
then nulls out the marker which means you'll continue to get the
marker instance back in the results even though you shouldn't,
and that can cause an infinite loop in some client-side tooling like
nova's CLI:
nova list --deleted --limit -1
This adds a functional recreate test for the regression which will
be updated when the bug is fixed.
Change-Id: I324193129acb6a
Related-Bug: #1849409
Related fix proposed to branch: stable/train
Review: https:/
Fix proposed to branch: stable/train
Review: https:/
Related fix proposed to branch: stable/stein
Review: https:/
Fix proposed to branch: stable/stein
Review: https:/
Related fix proposed to branch: stable/rocky
Review: https:/
Fix proposed to branch: stable/rocky
Review: https:/
Related fix proposed to branch: stable/queens
Review: https:/
Fix proposed to branch: stable/queens
Review: https:/
Related fix proposed to branch: stable/pike
Review: https:/
Fix proposed to branch: stable/pike
Review: https:/
Related fix proposed to branch: stable/ocata
Review: https:/
Fix proposed to branch: stable/ocata
Review: https:/
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: master
commit df03499843aa7fd
Author: Matt Riedemann <email address hidden>
Date: Tue Oct 22 17:59:32 2019 -0400
Fix listing deleted servers with a marker
Change I1aa3ca6cc70cef
which was backported through to Newton, introduced a regression
when listing deleted servers with a marker because it assumes
that if BuildRequestLis
MarkerNotFound that the marker was found among the build requests
and does not account for that get_by_filters method short-circuiting
if filtering servers with deleted/
then nulls out the marker which means you'll continue to get the
marker instance back in the results even though you shouldn't,
and that can cause an infinite loop in some client-side tooling like
nova's CLI:
nova list --deleted --limit -1
This fixes the bug by raising MarkerNotFound from
BuildReques
short-
limit or filters.
Change-Id: Ic2b19c2aa06b30
Closes-Bug: #1849409
| Changed in nova: | |
| status: | In Progress → Fix Released |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/train
commit 727d942b2830fb6
Author: Matt Riedemann <email address hidden>
Date: Tue Oct 22 17:53:20 2019 -0400
Add functional regression test for bug 1849409
Change I1aa3ca6cc70cef
which was backported through to Newton, introduced a regression
when listing deleted servers with a marker because it assumes
that if BuildRequestLis
MarkerNotFound that the marker was found among the build requests
and does not account for that get_by_filters method short-circuiting
if filtering servers with deleted/
then nulls out the marker which means you'll continue to get the
marker instance back in the results even though you shouldn't,
and that can cause an infinite loop in some client-side tooling like
nova's CLI:
nova list --deleted --limit -1
This adds a functional recreate test for the regression which will
be updated when the bug is fixed.
Change-Id: I324193129acb6a
Related-Bug: #1849409
(cherry picked from commit 45c2752f2ce08b3
| tags: | added: in-stable-train |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/train
commit 03a2508362ecf50
Author: Matt Riedemann <email address hidden>
Date: Tue Oct 22 17:59:32 2019 -0400
Fix listing deleted servers with a marker
Change I1aa3ca6cc70cef
which was backported through to Newton, introduced a regression
when listing deleted servers with a marker because it assumes
that if BuildRequestLis
MarkerNotFound that the marker was found among the build requests
and does not account for that get_by_filters method short-circuiting
if filtering servers with deleted/
then nulls out the marker which means you'll continue to get the
marker instance back in the results even though you shouldn't,
and that can cause an infinite loop in some client-side tooling like
nova's CLI:
nova list --deleted --limit -1
This fixes the bug by raising MarkerNotFound from
BuildReques
short-
limit or filters.
Change-Id: Ic2b19c2aa06b30
Closes-Bug: #1849409
(cherry picked from commit df03499843aa7fd
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/stein
commit 47caaccd4a03660
Author: Matt Riedemann <email address hidden>
Date: Tue Oct 22 17:53:20 2019 -0400
Add functional regression test for bug 1849409
Change I1aa3ca6cc70cef
which was backported through to Newton, introduced a regression
when listing deleted servers with a marker because it assumes
that if BuildRequestLis
MarkerNotFound that the marker was found among the build requests
and does not account for that get_by_filters method short-circuiting
if filtering servers with deleted/
then nulls out the marker which means you'll continue to get the
marker instance back in the results even though you shouldn't,
and that can cause an infinite loop in some client-side tooling like
nova's CLI:
nova list --deleted --limit -1
This adds a functional recreate test for the regression which will
be updated when the bug is fixed.
Change-Id: I324193129acb6a
Related-Bug: #1849409
(cherry picked from commit 45c2752f2ce08b3
(cherry picked from commit 727d942b2830fb6
| tags: | added: in-stable-stein |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/stein
commit 6038455e1dafeaf
Author: Matt Riedemann <email address hidden>
Date: Tue Oct 22 17:59:32 2019 -0400
Fix listing deleted servers with a marker
Change I1aa3ca6cc70cef
which was backported through to Newton, introduced a regression
when listing deleted servers with a marker because it assumes
that if BuildRequestLis
MarkerNotFound that the marker was found among the build requests
and does not account for that get_by_filters method short-circuiting
if filtering servers with deleted/
then nulls out the marker which means you'll continue to get the
marker instance back in the results even though you shouldn't,
and that can cause an infinite loop in some client-side tooling like
nova's CLI:
nova list --deleted --limit -1
This fixes the bug by raising MarkerNotFound from
BuildReques
short-
limit or filters.
Change-Id: Ic2b19c2aa06b30
Closes-Bug: #1849409
(cherry picked from commit df03499843aa7fd
(cherry picked from commit 03a2508362ecf50
This issue was fixed in the openstack/nova 20.0.1 release.
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/rocky
commit 08337cccb060d0b
Author: Matt Riedemann <email address hidden>
Date: Tue Oct 22 17:53:20 2019 -0400
Add functional regression test for bug 1849409
Change I1aa3ca6cc70cef
which was backported through to Newton, introduced a regression
when listing deleted servers with a marker because it assumes
that if BuildRequestLis
MarkerNotFound that the marker was found among the build requests
and does not account for that get_by_filters method short-circuiting
if filtering servers with deleted/
then nulls out the marker which means you'll continue to get the
marker instance back in the results even though you shouldn't,
and that can cause an infinite loop in some client-side tooling like
nova's CLI:
nova list --deleted --limit -1
This adds a functional recreate test for the regression which will
be updated when the bug is fixed.
NOTE(mriedem): In this backport we get the PlacementFixture from
the usual nova_fixtures import since change
Idaed396290
test_
servers from cell0 for some reason in Rocky which can make this test
intermittently fail. This is not a problem after Rocky for some reason.
To avoid intermittent failures due to that leak, the test is modified
to filter on the name of the server created in the test only.
Change-Id: I324193129acb6a
Related-Bug: #1849409
(cherry picked from commit 45c2752f2ce08b3
(cherry picked from commit 727d942b2830fb6
(cherry picked from commit 47caaccd4a03660
| tags: | added: in-stable-rocky |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/rocky
commit c84846dbf93588e
Author: Matt Riedemann <email address hidden>
Date: Tue Oct 22 17:59:32 2019 -0400
Fix listing deleted servers with a marker
Change I1aa3ca6cc70cef
which was backported through to Newton, introduced a regression
when listing deleted servers with a marker because it assumes
that if BuildRequestLis
MarkerNotFound that the marker was found among the build requests
and does not account for that get_by_filters method short-circuiting
if filtering servers with deleted/
then nulls out the marker which means you'll continue to get the
marker instance back in the results even though you shouldn't,
and that can cause an infinite loop in some client-side tooling like
nova's CLI:
nova list --deleted --limit -1
This fixes the bug by raising MarkerNotFound from
BuildReques
short-
limit or filters.
Conflicts:
nova/
NOTE(mriedem): The conflict is due to the Rocky-specific change
to the functional test in I324193129acb6a
due to test_concurrent
deleted servers for some reason.
Change-Id: Ic2b19c2aa06b30
Closes-Bug: #1849409
(cherry picked from commit df03499843aa7fd
(cherry picked from commit 03a2508362ecf50
(cherry picked from commit 6038455e1dafeaf
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/queens
commit f03f5075e3751cc
Author: Matt Riedemann <email address hidden>
Date: Tue Oct 22 17:53:20 2019 -0400
Add functional regression test for bug 1849409
Change I1aa3ca6cc70cef
which was backported through to Newton, introduced a regression
when listing deleted servers with a marker because it assumes
that if BuildRequestLis
MarkerNotFound that the marker was found among the build requests
and does not account for that get_by_filters method short-circuiting
if filtering servers with deleted/
then nulls out the marker which means you'll continue to get the
marker instance back in the results even though you shouldn't,
and that can cause an infinite loop in some client-side tooling like
nova's CLI:
nova list --deleted --limit -1
This adds a functional recreate test for the regression which will
be updated when the bug is fixed.
NOTE(mriedem): In this backport the test is modified to create the
server without networking to avoid a NetworkAmbiguous error
which is probably due to some missing NeutronFixture sugar added
in Rocky.
Change-Id: I324193129acb6a
Related-Bug: #1849409
(cherry picked from commit 45c2752f2ce08b3
(cherry picked from commit 727d942b2830fb6
(cherry picked from commit 47caaccd4a03660
(cherry picked from commit 08337cccb060d0b
| tags: | added: in-stable-queens |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/queens
commit 8c18293f93a0ef9
Author: Matt Riedemann <email address hidden>
Date: Tue Oct 22 17:59:32 2019 -0400
Fix listing deleted servers with a marker
Change I1aa3ca6cc70cef
which was backported through to Newton, introduced a regression
when listing deleted servers with a marker because it assumes
that if BuildRequestLis
MarkerNotFound that the marker was found among the build requests
and does not account for that get_by_filters method short-circuiting
if filtering servers with deleted/
then nulls out the marker which means you'll continue to get the
marker instance back in the results even though you shouldn't,
and that can cause an infinite loop in some client-side tooling like
nova's CLI:
nova list --deleted --limit -1
This fixes the bug by raising MarkerNotFound from
BuildReques
short-
limit or filters.
Change-Id: Ic2b19c2aa06b30
Closes-Bug: #1849409
(cherry picked from commit df03499843aa7fd
(cherry picked from commit 03a2508362ecf50
(cherry picked from commit 6038455e1dafeaf
(cherry picked from commit c84846dbf93588e
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/pike
commit e2ba87cb7b1cfd4
Author: Matt Riedemann <email address hidden>
Date: Tue Oct 22 17:53:20 2019 -0400
Add functional regression test for bug 1849409
Change I1aa3ca6cc70cef
which was backported through to Newton, introduced a regression
when listing deleted servers with a marker because it assumes
that if BuildRequestLis
MarkerNotFound that the marker was found among the build requests
and does not account for that get_by_filters method short-circuiting
if filtering servers with deleted/
then nulls out the marker which means you'll continue to get the
marker instance back in the results even though you shouldn't,
and that can cause an infinite loop in some client-side tooling like
nova's CLI:
nova list --deleted --limit -1
This adds a functional recreate test for the regression which will
be updated when the bug is fixed.
Change-Id: I324193129acb6a
Related-Bug: #1849409
(cherry picked from commit 45c2752f2ce08b3
(cherry picked from commit 727d942b2830fb6
(cherry picked from commit 47caaccd4a03660
(cherry picked from commit 08337cccb060d0b
(cherry picked from commit f03f5075e3751cc
| tags: | added: in-stable-pike |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/pike
commit b277ec154f2a175
Author: Matt Riedemann <email address hidden>
Date: Tue Oct 22 17:59:32 2019 -0400
Fix listing deleted servers with a marker
Change I1aa3ca6cc70cef
which was backported through to Newton, introduced a regression
when listing deleted servers with a marker because it assumes
that if BuildRequestLis
MarkerNotFound that the marker was found among the build requests
and does not account for that get_by_filters method short-circuiting
if filtering servers with deleted/
then nulls out the marker which means you'll continue to get the
marker instance back in the results even though you shouldn't,
and that can cause an infinite loop in some client-side tooling like
nova's CLI:
nova list --deleted --limit -1
This fixes the bug by raising MarkerNotFound from
BuildReques
short-
limit or filters.
Change-Id: Ic2b19c2aa06b30
Closes-Bug: #1849409
(cherry picked from commit df03499843aa7fd
(cherry picked from commit 03a2508362ecf50
(cherry picked from commit 6038455e1dafeaf
(cherry picked from commit c84846dbf93588e
(cherry picked from commit 8c18293f93a0ef9
This issue was fixed in the openstack/nova 19.1.0 release.
This issue was fixed in the openstack/nova 18.3.0 release.
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/ocata
commit b6c7456face6300
Author: Matt Riedemann <email address hidden>
Date: Tue Oct 22 17:53:20 2019 -0400
Add functional regression test for bug 1849409
Change I1aa3ca6cc70cef
which was backported through to Newton, introduced a regression
when listing deleted servers with a marker because it assumes
that if BuildRequestLis
MarkerNotFound that the marker was found among the build requests
and does not account for that get_by_filters method short-circuiting
if filtering servers with deleted/
then nulls out the marker which means you'll continue to get the
marker instance back in the results even though you shouldn't,
and that can cause an infinite loop in some client-side tooling like
nova's CLI:
nova list --deleted --limit -1
This adds a functional recreate test for the regression which will
be updated when the bug is fixed.
NOTE(mriedem): In this backport the test is modified to disable the
DiskFilter since we're using Placement for filtering on DISK_GB.
Also, _wait_until_deleted is moved to InstanceHelperMixin since
If7b02bcd8d
Change-Id: I324193129acb6a
Related-Bug: #1849409
(cherry picked from commit 45c2752f2ce08b3
(cherry picked from commit 727d942b2830fb6
(cherry picked from commit 47caaccd4a03660
(cherry picked from commit 08337cccb060d0b
(cherry picked from commit f03f5075e3751cc
(cherry picked from commit e2ba87cb7b1cfd4
| tags: | added: in-stable-ocata |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/ocata
commit b91592595141d77
Author: Matt Riedemann <email address hidden>
Date: Tue Oct 22 17:59:32 2019 -0400
Fix listing deleted servers with a marker
Change I1aa3ca6cc70cef
which was backported through to Newton, introduced a regression
when listing deleted servers with a marker because it assumes
that if BuildRequestLis
MarkerNotFound that the marker was found among the build requests
and does not account for that get_by_filters method short-circuiting
if filtering servers with deleted/
then nulls out the marker which means you'll continue to get the
marker instance back in the results even though you shouldn't,
and that can cause an infinite loop in some client-side tooling like
nova's CLI:
nova list --deleted --limit -1
This fixes the bug by raising MarkerNotFound from
BuildReques
short-
limit or filters.
Change-Id: Ic2b19c2aa06b30
Closes-Bug: #1849409
(cherry picked from commit df03499843aa7fd
(cherry picked from commit 03a2508362ecf50
(cherry picked from commit 6038455e1dafeaf
(cherry picked from commit c84846dbf93588e
(cherry picked from commit 8c18293f93a0ef9
(cherry picked from commit b277ec154f2a175
I've got the fix started here: https:/