nova servers pagination does not work with deleted marker

[Alvaro Lopez]

Nova does not paginate correctly if the marker is a deleted server.

I am trying to get all of the servers for a given tenant. In total (i.e. active, delete, error, etc.) there are 405 servers.

If I query the API without a marker and with a limit larger (for example, 500)
than the total number of servers I get all of them, i.e. the following query
correctly returns 405 servers:

    curl (...) "http://cloud.example.org:8774/v1.1/foo/servers?changes-since=2014-01-01&limit=500"

However, if I try to paginate over them, doing:

    curl (...) "http://cloud.example.org:8774/v1.1/foo/servers?changes-since=2014-01-01&limit=100"

I get the first 100 with a link to the next page. If I try to follow it:

    curl (...) "http://cloud.example.org:8774/v1.1/foo/servers?changes-since=2014-01-01&limit=100&marker=foobar"

I am always getting a "badRequest" error saying that the marker is not found. I
guess this is because of these lines in "nova/db/sqlalchemy/api.py"

    2000 # paginate query
    2001 if marker is not None:
    2002 try:
    2003 marker = _instance_get_by_uuid(context, marker, session=session)
    2004 except exception.InstanceNotFound:
    2005 raise exception.MarkerNotFound(marker)

The function "_instance_get_by_uuid" gets the machines that are not deleted, therefore it fails to locate the marker if it is a deleted server.

[Deliang Fan (vanderliang)]

In my opinion, it's not a bug because the deleted vm should not be queried.

[Alvaro Lopez (aloga)]

Sorry, but no, this is not invalid.

The API explicitly allows querying for machines in a given status (active, deleted, etc.): http://developer.openstack.org/api-ref-compute-v2.html#compute_servers

Lastly, in the case that it would not be possible, then the marker returned by nova is wrong, so the bug is still valid.

[Joe Gordon (jogo)]

Confirmed, if you have two deleted servers (test-server-1, test-server-2) they should appear in "nova list --status=deleted" and using pagination without a maker works, so "http://10.0.0.17:8774/v2/UUID/servers?status=deleted&limit=1" works. But

http://10.0.0.17:8774/v2/d0a91ae0a2b647acbb06805c677a65c1/servers?status=deleted&limit=1&marker=test-server-2

doesn't work. It returns :

"badRequest" : {
      "code" : 400,
      "message" : "marker [test-server-2] not found"
   }
"

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/139306

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/139306
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d85bd3e748ea17d523d011387dcf8a5af0318c20
Submitter: Jenkins
Branch: master

commit d85bd3e748ea17d523d011387dcf8a5af0318c20
Author: liyingjun <email address hidden>
Date: Fri Dec 5 10:34:56 2014 +0800

    Make pagination work with deleted marker

    Currently, Nova does not paginate correctly if the marker is a
    deleted server. When trying to get servers with deleted and marker,
    a "BadRequest" error saying that the marker is not found will be
    raised.

    Fixes the bug by passing read_deleted to context when getting
    server to enable to get deleted server as marker.

    Change-Id: Icde4e7d251d4497ee8eeb0a573eca3b73aa71466
    Closes-bug: #1398086

[Christopher Armstrong (radix)]

I don't think this bug, as originally reported, was fixed. The fix only addressed the use of `status=deleted`, but the original report is for use of `changes-since`, which is supposed to implicitly include deleted servers. The bug is still occurring.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/209554

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/212614

[OpenStack Infra (hudson-openstack)]

Change abandoned by Martin Schuppert (<email address hidden>) on branch: master
Review: https://review.openstack.org/212614
Reason: sorry, mistake in submit the commit

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/209554
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=901c223f351310b38fa17aa1c3dcd758ce897e5e
Submitter: Jenkins
Branch: master

commit 901c223f351310b38fa17aa1c3dcd758ce897e5e
Author: Martin Schuppert <email address hidden>
Date: Wed Aug 5 17:10:14 2015 +0200

    Make pagination tolerate a deleted marker

    Currently, Nova does not paginate correctly if the marker is
    a deleted server. When trying to get servers with such a
    marker, a "BadRequest" error saying that the marker is
    not found will be raised.

    We now always pass read_deleted=yes if a marker is being used.

    Change-Id: I79a1fb79a38ac4006db622590574308aafdcb5b5
    Closes-bug: #1481812
    Related-Bug: #1398086

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/kilo
Review: https://review.openstack.org/214986

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/214986
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8fccffda596e7f332f7f4a440bd75c8607a7ef14
Submitter: Jenkins
Branch: stable/kilo

commit 8fccffda596e7f332f7f4a440bd75c8607a7ef14
Author: Martin Schuppert <email address hidden>
Date: Wed Aug 5 17:10:14 2015 +0200

    Make pagination tolerate a deleted marker

    Currently, Nova does not paginate correctly if the marker is
    a deleted server. When trying to get servers with such a
    marker, a "BadRequest" error saying that the marker is
    not found will be raised.

    We now always pass read_deleted=yes if a marker is being used.

    Change-Id: I79a1fb79a38ac4006db622590574308aafdcb5b5
    Closes-bug: #1481812
    Related-Bug: #1398086
    (cherry picked from commit 901c223f351310b38fa17aa1c3dcd758ce897e5e)