nova servers pagination does not work with changes-since and deleted marker

[Martin Schuppert]

instance test1 - test6, where test2 and test5 has been deleted:

# nova list
+--------------------------------------+-------+---------+------------+-------------+------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+---------+------------+-------------+------------------+
| 7e12d6a0-126f-44d0-b566-15cd5e4ab82e | test1 | SHUTOFF | - | Shutdown | private=10.0.0.3 |
| 8b35f7fb-65d0-4fc3-ac22-390743c695db | test3 | ACTIVE | - | Running | private=10.0.0.5 |
| 2ab70dfe-2983-4886-a930-7deb15279763 | test4 | ACTIVE | - | Running | private=10.0.0.6 |
| 489e22cf-5e22-43a4-8c46-438f62d66e59 | test6 | ACTIVE | - | Running | private=10.0.0.8 |
+--------------------------------------+-------+---------+------------+-------------+------------------+

# Get all instances with changes-since=2015-01-01 :
# curl -s -H "X-Auth-Token:92ecba357e5b49f88a21cedfa63bf36e" 'http://10.10.180.210:8774/v2/30d2b54aa8f64bc2a1577c992c16271a/servers?changes-since=2015-01-01';
{"servers": [{"id": "489e22cf-5e22-43a4-8c46-438f62d66e59", "links": [{"href": "http://10.10.180.210:8774/v2/30d2b54aa8f64bc2a1577c992c16271a/servers/489e22cf-5e22-43a4-8c46-438f62d66e59", "rel": "self"}, {"href": "http://10.10.180.210:8774/30d2b54aa8f64bc2a1577c992c16271a/servers/489e22cf-5e22-43a4-8c46-438f62d66e59", "rel": "bookmark"}], "name": "test6"}, {"id": "9bda60eb-6ff7-4b84-b081-0120b62155a3", "links": [{"href": "http://10.10.180.210:8774/v2/30d2b54aa8f64bc2a1577c992c16271a/servers/9bda60eb-6ff7-4b84-b081-0120b62155a3", "rel": "self"}, {"href": "http://10.10.180.210:8774/30d2b54aa8f64bc2a1577c992c16271a/servers/9bda60eb-6ff7-4b84-b081-0120b62155a3", "rel": "bookmark"}], "name": "test5"}, {"id": "2ab70dfe-2983-4886-a930-7deb15279763", "links": [{"href": "http://10.10.180.210:8774/v2/30d2b54aa8f64bc2a1577c992c16271a/servers/2ab70dfe-2983-4886-a930-7deb15279763", "rel": "self"}, {"href": "http://10.10.180.210:8774/30d2b54aa8f64bc2a1577c992c16271a/servers/2ab70dfe-2983-4886-a930-7deb15279763", "rel": "bookmark"}], "name": "test4"}, {"id": "8b35f7fb-65d0-4fc3-ac22-390743c695db", "links": [{"href": "http://10.10.180.210:8774/v2/30d2b54aa8f64bc2a1577c992c16271a/servers/8b35f7fb-65d0-4fc3-ac22-390743c695db", "rel": "self"}, {"href": "http://10.10.180.210:8774/30d2b54aa8f64bc2a1577c992c16271a/servers/8b35f7fb-65d0-4fc3-ac22-390743c695db", "rel": "bookmark"}], "name": "test3"}, {"id": "18d9ffbb-e1d4-4218-bb66-f792aab4e091", "links": [{"href": "http://10.10.180.210:8774/v2/30d2b54aa8f64bc2a1577c992c16271a/servers/18d9ffbb-e1d4-4218-bb66-f792aab4e091", "rel": "self"}, {"href": "http://10.10.180.210:8774/30d2b54aa8f64bc2a1577c992c16271a/servers/18d9ffbb-e1d4-4218-bb66-f792aab4e091", "rel": "bookmark"}], "name": "test2"}, {"id": "7e12d6a0-126f-44d0-b566-15cd5e4ab82e", "links": [{"href": "http://10.10.180.210:8774/v2/30d2b54aa8f64bc2a1577c992c16271a/servers/7e12d6a0-126f-44d0-b566-15cd5e4ab82e", "rel": "self"}, {"href": "http://10.10.180.210:8774/30d2b54aa8f64bc2a1577c992c16271a/servers/7e12d6a0-126f-44d0-b566-15cd5e4ab82e", "rel": "bookmark"}], "name": "test1"}]}

# query the instances in junks of 2 with changes-since and limit=2

# curl -s -H "X-Auth-Token:92ecba357e5b49f88a21cedfa63bf36e" 'http://10.10.180.210:8774/v2/30d2b54aa8f64bc2a1577c992c16271a/servers?changes-since=2015-01-01&limit=2';
{"servers_links": [{"href": "http://10.10.180.210:8774/v2/30d2b54aa8f64bc2a1577c992c16271a/servers?changes-since=2015-01-01&limit=2&marker=9bda60eb-6ff7-4b84-b081-0120b62155a3", "rel": "next"}], "servers": [{"id": "489e22cf-5e22-43a4-8c46-438f62d66e59", "links": [{"href": "http://10.10.180.210:8774/v2/30d2b54aa8f64bc2a1577c992c16271a/servers/489e22cf-5e22-43a4-8c46-438f62d66e59", "rel": "self"}, {"href": "http://10.10.180.210:8774/30d2b54aa8f64bc2a1577c992c16271a/servers/489e22cf-5e22-43a4-8c46-438f62d66e59", "rel": "bookmark"}], "name": "test6"}, {"id": "9bda60eb-6ff7-4b84-b081-0120b62155a3", "links": [{"href": "http://10.10.180.210:8774/v2/30d2b54aa8f64bc2a1577c992c16271a/servers/9bda60eb-6ff7-4b84-b081-0120b62155a3", "rel": "self"}, {"href": "http://10.10.180.210:8774/30d2b54aa8f64bc2a1577c992c16271a/servers/9bda60eb-6ff7-4b84-b081-0120b62155a3", "rel": "bookmark"}], "name": "test5"}]}

=> returns instance test6 and test5(deleted)

# use instance test5 (deleted) from previous query as a marker to get the next 2:

# curl -s -H "X-Auth-Token:92ecba357e5b49f88a21cedfa63bf36e" 'http://10.10.180.210:8774/v2/30d2b54aa8f64bc2a1577c992c16271a/servers?changes-since=2015-01-01&limit=2&marker=9bda60eb-6ff7-4b84-b081-0120b62155a3';
{"badRequest": {"message": "marker [9bda60eb-6ff7-4b84-b081-0120b62155a3] not found", "code": 400}}

=> marker not found!

Related to https://bugs.launchpad.net/nova/+bug/1398086 where fix has been submitted to resolve status=deleted&marker=X

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/209554

[Roman Podoliaka (rpodolyaka)]

Hmm, looks like we forbid users to list deleted instances (when done by simple "nova list"), but at the same time passing "changes-since" allows to circumvent that.

ubuntu@rpodolyaka-devstack-tempest:~/devstack$ nova list --deleted
+--------------------------------------+------+--------+------------+-------------+--------------------------------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+--------------------------------------------------------+
| 0fc22f61-a55e-46de-87fd-94a1e7b2141c | demo | ACTIVE | - | Running | private=10.0.0.5, fdad:2959:8cd7:0:f816:3eff:fe04:522d |
+--------------------------------------+------+--------+------------+-------------+--------------------------------------------------------+

ubuntu@rpodolyaka-devstack-tempest:~/devstack$ curl -s -H "X-Auth-Token: e9819485d45f428c8e52c03b22460cc9" -X GET http://192.168.0.13:8774/v2/dc3cad5a802b4ba1b86f8562d2f91c0d/servers?changes-since=2015-01-01\&limit=2 | python -m json.tool
{
    "servers": [
        {
            "id": "0fc22f61-a55e-46de-87fd-94a1e7b2141c",
            "links": [
                {
                    "href": "http://192.168.0.13:8774/v2/dc3cad5a802b4ba1b86f8562d2f91c0d/servers/0fc22f61-a55e-46de-87fd-94a1e7b2141c",
                    "rel": "self"
                },
                {
                    "href": "http://192.168.0.13:8774/dc3cad5a802b4ba1b86f8562d2f91c0d/servers/0fc22f61-a55e-46de-87fd-94a1e7b2141c",
                    "rel": "bookmark"
                }
            ],
            "name": "demo"
        },
        {
            "id": "7c837fc6-3503-4ae0-b474-1b1e55ca446a",
            "links": [
                {
                    "href": "http://192.168.0.13:8774/v2/dc3cad5a802b4ba1b86f8562d2f91c0d/servers/7c837fc6-3503-4ae0-b474-1b1e55ca446a",
                    "rel": "self"
                },
                {
                    "href": "http://192.168.0.13:8774/dc3cad5a802b4ba1b86f8562d2f91c0d/servers/7c837fc6-3503-4ae0-b474-1b1e55ca446a",
                    "rel": "bookmark"
                }
            ],
            "name": "tempest-heat-319343001-Server-f5bl4lw4ejln"
        }
    ],
    "servers_links": [
        {
            "href": "http://192.168.0.13:8774/v2/dc3cad5a802b4ba1b86f8562d2f91c0d/servers?changes-since=2015-01-01&limit=2&marker=7c837fc6-3503-4ae0-b474-1b1e55ca446a",
            "rel": "next"
        }
    ]
}

Still, when marker is passed, the deleted entry will be invisible to the user and cause the error.

[Martin Schuppert (mschuppert)]

This issue is not just for non admin users. It also fails for admin users who can query the deleted instances and it fails when using the marker for a deleted instance as well.

This is related to what has been already merged in https://bugs.launchpad.net/nova/+bug/1398086 / https://review.openstack.org/#/c/139306/ when status=deleted&marker= is being used.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/212614

[OpenStack Infra (hudson-openstack)]

Change abandoned by Martin Schuppert (<email address hidden>) on branch: master
Review: https://review.openstack.org/212614
Reason: sorry, mistake in submit the commit

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/209554
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=901c223f351310b38fa17aa1c3dcd758ce897e5e
Submitter: Jenkins
Branch: master

commit 901c223f351310b38fa17aa1c3dcd758ce897e5e
Author: Martin Schuppert <email address hidden>
Date: Wed Aug 5 17:10:14 2015 +0200

    Make pagination tolerate a deleted marker

    Currently, Nova does not paginate correctly if the marker is
    a deleted server. When trying to get servers with such a
    marker, a "BadRequest" error saying that the marker is
    not found will be raised.

    We now always pass read_deleted=yes if a marker is being used.

    Change-Id: I79a1fb79a38ac4006db622590574308aafdcb5b5
    Closes-bug: #1481812
    Related-Bug: #1398086

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/214986

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/214986
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8fccffda596e7f332f7f4a440bd75c8607a7ef14
Submitter: Jenkins
Branch: stable/kilo

commit 8fccffda596e7f332f7f4a440bd75c8607a7ef14
Author: Martin Schuppert <email address hidden>
Date: Wed Aug 5 17:10:14 2015 +0200

    Make pagination tolerate a deleted marker

    Currently, Nova does not paginate correctly if the marker is
    a deleted server. When trying to get servers with such a
    marker, a "BadRequest" error saying that the marker is
    not found will be raised.

    We now always pass read_deleted=yes if a marker is being used.

    Change-Id: I79a1fb79a38ac4006db622590574308aafdcb5b5
    Closes-bug: #1481812
    Related-Bug: #1398086
    (cherry picked from commit 901c223f351310b38fa17aa1c3dcd758ce897e5e)