Listing instances with a marker doesn't nix the marker if it's found in build_requests

[Matt Riedemann]

When listing instances, we start with the build requests and then hit the cells.

If we're given a marker, we use it to trim the build_requests:

https://github.com/openstack/nova/blob/master/nova/objects/build_request.py#L440-L457

But normally if you're looking for a marker and don't find it, that get_by_filters code should raise MarkerNotFound to indicate to the caller that you asked to filter with a marker which isn't here.

If we got results back from build_requests with a marker, then the compute API get_all() method should null out the marker and continue filling the limit in the cells, like what we'd do here:

https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2426

And this is how it's handled within a cell database:

https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L2242-L2257

[Matt Riedemann (mriedem)]

This code was introduced in https://review.openstack.org/#/c/356137/ in newton so we'd have to backport that far if we can before newton end of life.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/527564

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/527564
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1706e3989157f912bce99beb99c75216a064eb2d
Submitter: Zuul
Branch: master

commit 1706e3989157f912bce99beb99c75216a064eb2d
Author: Matt Riedemann <email address hidden>
Date: Tue Dec 12 21:27:28 2017 -0500

    Raise MarkerNotFound if BuildRequestList.get_by_filters doesn't find marker

    For some reason, probably because build requests are meant to be short lived
    and we don't get a lot of bugs about paging misbehavior, when paging instances
    with a marker, we didn't raise MarkerNotFound if we didn't find the marker in
    the list of build requests. Doing so would match what we do when paging over
    cells and listing instances using a marker. Once we find the marker, be that
    in build_requests, or one of the cells, we need to set the marker to None to
    stop looking for it elsewhere if we have more space to fill our limit.

    For example, see change I8a957bebfcecd6ac712103c346e028d80f1ecd7c.

    This patch fixes the issue by raising MarkerNotFound from BuildRequestList
    get_by_filters if there is a marker and we didn't find a build request for
    it. The compute API get_all() method handles that as normal and continues
    looking for the marker in one of the cells.

    Change-Id: I1aa3ca6cc70cef65d24dec1e7db9491c9b73f7ab
    Closes-Bug: #1737856

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/530980

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/530981

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/530982

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/530980
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=344029b94ad7ff4667403a851d33c3ddb4e97b4b
Submitter: Zuul
Branch: stable/pike

commit 344029b94ad7ff4667403a851d33c3ddb4e97b4b
Author: Matt Riedemann <email address hidden>
Date: Tue Dec 12 21:27:28 2017 -0500

    Raise MarkerNotFound if BuildRequestList.get_by_filters doesn't find marker

    For some reason, probably because build requests are meant to be short lived
    and we don't get a lot of bugs about paging misbehavior, when paging instances
    with a marker, we didn't raise MarkerNotFound if we didn't find the marker in
    the list of build requests. Doing so would match what we do when paging over
    cells and listing instances using a marker. Once we find the marker, be that
    in build_requests, or one of the cells, we need to set the marker to None to
    stop looking for it elsewhere if we have more space to fill our limit.

    For example, see change I8a957bebfcecd6ac712103c346e028d80f1ecd7c.

    This patch fixes the issue by raising MarkerNotFound from BuildRequestList
    get_by_filters if there is a marker and we didn't find a build request for
    it. The compute API get_all() method handles that as normal and continues
    looking for the marker in one of the cells.

    Conflicts:
          nova/tests/unit/compute/test_compute_api.py

    NOTE(mriedem): The conflicts are due to not having
    3e37c2e10e1701c4cc782f31b9e5917011123261 nor
    d5244f2fae1e0b0c6e80c1e3b2f1b48fd2c8d1d9 in Pike.

    Change-Id: I1aa3ca6cc70cef65d24dec1e7db9491c9b73f7ab
    Closes-Bug: #1737856
    (cherry picked from commit 1706e3989157f912bce99beb99c75216a064eb2d)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/530981
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b00b2fe9be4394e7c9cf73c168d435a2333d12f6
Submitter: Zuul
Branch: stable/ocata

commit b00b2fe9be4394e7c9cf73c168d435a2333d12f6
Author: Matt Riedemann <email address hidden>
Date: Tue Dec 12 21:27:28 2017 -0500

    Raise MarkerNotFound if BuildRequestList.get_by_filters doesn't find marker

    For some reason, probably because build requests are meant to be short lived
    and we don't get a lot of bugs about paging misbehavior, when paging instances
    with a marker, we didn't raise MarkerNotFound if we didn't find the marker in
    the list of build requests. Doing so would match what we do when paging over
    cells and listing instances using a marker. Once we find the marker, be that
    in build_requests, or one of the cells, we need to set the marker to None to
    stop looking for it elsewhere if we have more space to fill our limit.

    For example, see change I8a957bebfcecd6ac712103c346e028d80f1ecd7c.

    This patch fixes the issue by raising MarkerNotFound from BuildRequestList
    get_by_filters if there is a marker and we didn't find a build request for
    it. The compute API get_all() method handles that as normal and continues
    looking for the marker in one of the cells.

    Conflicts:
          nova/tests/unit/compute/test_compute_api.py

    NOTE(mriedem): The conflicts are due to not having
    bb7c6ebb5648150430363c12ebc25a1c405ca79f in Ocata.

    Change-Id: I1aa3ca6cc70cef65d24dec1e7db9491c9b73f7ab
    Closes-Bug: #1737856
    (cherry picked from commit 1706e3989157f912bce99beb99c75216a064eb2d)
    (cherry picked from commit 344029b94ad7ff4667403a851d33c3ddb4e97b4b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/530982
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=808d36475103e373f1deb3344b6829ce68d6cdd5
Submitter: Zuul
Branch: stable/newton

commit 808d36475103e373f1deb3344b6829ce68d6cdd5
Author: Matt Riedemann <email address hidden>
Date: Tue Dec 12 21:27:28 2017 -0500

    Raise MarkerNotFound if BuildRequestList.get_by_filters doesn't find marker

    For some reason, probably because build requests are meant to be short lived
    and we don't get a lot of bugs about paging misbehavior, when paging instances
    with a marker, we didn't raise MarkerNotFound if we didn't find the marker in
    the list of build requests. Doing so would match what we do when paging over
    cells and listing instances using a marker. Once we find the marker, be that
    in build_requests, or one of the cells, we need to set the marker to None to
    stop looking for it elsewhere if we have more space to fill our limit.

    For example, see change I8a957bebfcecd6ac712103c346e028d80f1ecd7c.

    This patch fixes the issue by raising MarkerNotFound from BuildRequestList
    get_by_filters if there is a marker and we didn't find a build request for
    it. The compute API get_all() method handles that as normal and continues
    looking for the marker in one of the cells.

    Change-Id: I1aa3ca6cc70cef65d24dec1e7db9491c9b73f7ab
    Closes-Bug: #1737856
    (cherry picked from commit 1706e3989157f912bce99beb99c75216a064eb2d)
    (cherry picked from commit 344029b94ad7ff4667403a851d33c3ddb4e97b4b)
    (cherry picked from commit b00b2fe9be4394e7c9cf73c168d435a2333d12f6)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 14.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 15.1.1 release.