OpenStack Compute (nova)

RequestSpec online data migration context does not contain project_id

Bug #1739318 reported by Mohammed Naser
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Matt Riedemann
Newton
Fix Released
High
Matt Riedemann
Ocata
Fix Released
High
Matt Riedemann
Pike
Fix Released
High
Matt Riedemann

Bug Description

The online data migration generates a context in order to be able to execute migrations:

https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L747

However, this context does not contain a `project_id` when running this via CLI.

https://github.com/openstack/nova/blob/master/nova/context.py#L279-L290

During the creation of RequestSpec's for old instances, the context which contains no `project_id`.

https://github.com/openstack/nova/blob/master/nova/objects/request_spec.py#L611-L622

This means that a RequestSpec gets created with `project_id` set to `null`. During the day-to-day operations, things work okay, however, when attempting to do a live migration, the `project_id` is set to `null` when trying to claim resources which the placement API refuses.

https://github.com/openstack/nova/blob/master/nova/scheduler/utils.py#L791

This will give errors as such:

<html>
 <head>
  <title>400 Bad Request</title>
 </head>
 <body>
  <h1>400 Bad Request</h1>
  The server could not comply with the request since it is either malformed or otherwise incorrect.<br /><br />
JSON does not validate: None is not of type 'string'

Failed validating 'type' in schema['properties']['project_id']:
    {'maxLength': 255, 'minLength': 1, 'type': 'string'}

On instance['project_id']:
    None

 </body>
</html>

Matt Riedemann (mriedem)
Changed in nova:
status: New → Triaged
importance: Undecided → High
assignee: nobody → Matt Riedemann (mriedem)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/529184

Changed in nova:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/529185

Matt Riedemann (mriedem)
summary: - Online data migration context does not contain project_id
+ RequestSpec online data migration context does not contain project_id
tags: added: up
tags: added: upgrade
removed: up
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/529384

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/529385

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/529387

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/529389

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/529184
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a148e638dd5eaaa6f9c8749782a625c6523b6b6a
Submitter: Zuul
Branch: master

commit a148e638dd5eaaa6f9c8749782a625c6523b6b6a
Author: Matt Riedemann <email address hidden>
Date: Tue Dec 19 18:31:27 2017 -0500

    Use instance.project_id when creating request specs for old instances

    The online data migration routine to create request specs for old
    instances is using an admin context, which doesn't have a project_id
    set, which means the request spec that gets created doesn't have
    a project_id set, which matters when the FilterScheduler tries to
    claim resources (create allocations in Placement) using said
    request spec's project_id (PUT /allocations requires a non-null
    project_id).

    This fixes the online data migration by creating the request spec
    using the instance's project_id rather than that from the context.

    Change-Id: I214a44f0eee7d90be5cd89f32f6e0017b19a3fd6
    Partial-Bug: #1739318

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/529185
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=f9a06c42535ef7ebba92c793afd947073352f850
Submitter: Zuul
Branch: master

commit f9a06c42535ef7ebba92c793afd947073352f850
Author: Matt Riedemann <email address hidden>
Date: Tue Dec 19 18:40:31 2017 -0500

    Workaround missing RequestSpec.project_id when moving an instance

    The online data migration routine to create request specs for old
    instances used an admin context which has an empty project_id,
    so when scheduling (moving) one of these, if we try to PUT /allocations
    in placement using the FilterScheduler we'll fail because the project_id
    is None.

    This works around that by putting the instance.project_id on the request
    spec before calling the scheduler to pick a node and claim resources
    against it.

    A later change will need to add some sort of online data migration
    routine so that we properly update and persist the fix for these
    older records.

    Change-Id: I34b1d99a9d0d2aca80f094a79ec1656abaf762dc
    Partial-Bug: #1739318

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/ocata)

Reviewed: https://review.openstack.org/529387
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ff2231f58389fc1a1f7c7b6fbf66ff0602a5cbfc
Submitter: Zuul
Branch: stable/ocata

commit ff2231f58389fc1a1f7c7b6fbf66ff0602a5cbfc
Author: Matt Riedemann <email address hidden>
Date: Tue Dec 19 18:31:27 2017 -0500

    Use instance.project_id when creating request specs for old instances

    The online data migration routine to create request specs for old
    instances is using an admin context, which doesn't have a project_id
    set, which means the request spec that gets created doesn't have
    a project_id set, which matters when the FilterScheduler tries to
    claim resources (create allocations in Placement) using said
    request spec's project_id (PUT /allocations requires a non-null
    project_id).

    This fixes the online data migration by creating the request spec
    using the instance's project_id rather than that from the context.

    Conflicts:
          nova/objects/request_spec.py

    NOTE(mriedem): The conflict is due to
    e211fca55a11c80058d5d78e31dc3ad466d7edfd which is not in
    Ocata nor is it needed.

    Change-Id: I214a44f0eee7d90be5cd89f32f6e0017b19a3fd6
    Partial-Bug: #1739318
    (cherry picked from commit a148e638dd5eaaa6f9c8749782a625c6523b6b6a)
    (cherry picked from commit c86db10feb07d023b651fd9909e1039fe775424d)

tags: added: in-stable-ocata
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/newton)

Reviewed: https://review.openstack.org/529389
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=99b9cff2e49c42b4ba7fbaad8393fd89268f1f6b
Submitter: Zuul
Branch: stable/newton

commit 99b9cff2e49c42b4ba7fbaad8393fd89268f1f6b
Author: Matt Riedemann <email address hidden>
Date: Tue Dec 19 18:31:27 2017 -0500

    Use instance.project_id when creating request specs for old instances

    The online data migration routine to create request specs for old
    instances is using an admin context, which doesn't have a project_id
    set, which means the request spec that gets created doesn't have
    a project_id set, which matters when the FilterScheduler tries to
    claim resources (create allocations in Placement) using said
    request spec's project_id (PUT /allocations requires a non-null
    project_id).

    This fixes the online data migration by creating the request spec
    using the instance's project_id rather than that from the context.

    Conflicts:
          nova/objects/request_spec.py

    NOTE(mriedem): The conflict is due to
    b26ef568077f1bd9af8076c362bad06da3ec7e43 which is not
    in Newton nor is it needed.

    Change-Id: I214a44f0eee7d90be5cd89f32f6e0017b19a3fd6
    Partial-Bug: #1739318
    (cherry picked from commit a148e638dd5eaaa6f9c8749782a625c6523b6b6a)
    (cherry picked from commit c86db10feb07d023b651fd9909e1039fe775424d)
    (cherry picked from commit ff2231f58389fc1a1f7c7b6fbf66ff0602a5cbfc)

tags: added: in-stable-newton
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/pike)

Reviewed: https://review.openstack.org/529384
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c86db10feb07d023b651fd9909e1039fe775424d
Submitter: Zuul
Branch: stable/pike

commit c86db10feb07d023b651fd9909e1039fe775424d
Author: Matt Riedemann <email address hidden>
Date: Tue Dec 19 18:31:27 2017 -0500

    Use instance.project_id when creating request specs for old instances

    The online data migration routine to create request specs for old
    instances is using an admin context, which doesn't have a project_id
    set, which means the request spec that gets created doesn't have
    a project_id set, which matters when the FilterScheduler tries to
    claim resources (create allocations in Placement) using said
    request spec's project_id (PUT /allocations requires a non-null
    project_id).

    This fixes the online data migration by creating the request spec
    using the instance's project_id rather than that from the context.

    Change-Id: I214a44f0eee7d90be5cd89f32f6e0017b19a3fd6
    Partial-Bug: #1739318
    (cherry picked from commit a148e638dd5eaaa6f9c8749782a625c6523b6b6a)

tags: added: in-stable-pike
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/529385
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7f8a14dbc88f8bb99d6a9da8eab8b16b067603ee
Submitter: Zuul
Branch: stable/pike

commit 7f8a14dbc88f8bb99d6a9da8eab8b16b067603ee
Author: Matt Riedemann <email address hidden>
Date: Tue Dec 19 18:40:31 2017 -0500

    Workaround missing RequestSpec.project_id when moving an instance

    The online data migration routine to create request specs for old
    instances used an admin context which has an empty project_id,
    so when scheduling (moving) one of these, if we try to PUT /allocations
    in placement using the FilterScheduler we'll fail because the project_id
    is None.

    This works around that by putting the instance.project_id on the request
    spec before calling the scheduler to pick a node and claim resources
    against it.

    A later change will need to add some sort of online data migration
    routine so that we properly update and persist the fix for these
    older records.

    Conflicts:
          nova/conductor/manager.py
          nova/conductor/tasks/live_migrate.py
          nova/conductor/tasks/migrate.py
          nova/tests/unit/conductor/test_conductor.py

    NOTE(mriedem): The conflicts are due to
    ca716ce4dd512e864886998a24e111e7d6e05848 and
    685c16041c7e84fb0861b9b1833fc6c3cc372d05 not
    being in Pike.

    Change-Id: I34b1d99a9d0d2aca80f094a79ec1656abaf762dc
    Partial-Bug: #1739318
    (cherry picked from commit f9a06c42535ef7ebba92c793afd947073352f850)

Revision history for this message
Matt Riedemann (mriedem) wrote :

I've closed this bug since the immediate issue is fixed. In order to remove the workaround during move operations from change I34b1d99a9d0d2aca80f094a79ec1656abaf762dc we'd have to add an online data migration, but that could be done later separately from this bug.

Changed in nova:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.