Nova does not ensure a valid token is available if snapshot process exceeds token lifetime

[Micheal Jones]

Recently we encountered the following issue due to the change in Icehouse for the default lifetime of a token before it expires. It's now 1 hour, while previously it was 8.

If a snapshot process takes longer than an hour, when it goes to the next phase it will fail with a 401 Unauthorized error because it has an invalid token.

In our specific example the following would take place:

1. User would set a snapshot to begin and a token would be associated with this request.
2. Snapshot would be created, compression time would take about 55 minutes. Enough to just push the snapshotting of this instance over the 60 minute mark.
3. Upon Image Upload ("Uploading image data for image" in the logs) Nova would then return a 401 Unauthorized error stating "This server could not verify that you are authorized to access the document you requested. Either you supplied the wrong credentials (e.g., bad password), or your browser does not understand how to supply the credentials required."

Icehouse 2014.1.2, KVM as the hypervisor.

The workaround is to specify a longer token timeout - however limits the ability to set short token expirations.

A possible solution may be to get a new/refresh the token if the time has exceeded the timeout.

[Nikolay Starodubtsev (starodubcevna)]

This bug should be moved to glance or keystone, it's not nova problem

[Micheal Jones (micheal-s)]

For what it's worth I filed this against Nova as it's Nova making the calls (and in charge of the snapshot process) - as the issue is created before the image is even sent to Glance because of the expired token. In my example it's only when it's time to sent the finished image to Glance that the expired token issue is revealed.

[Nikolay Starodubtsev (starodubcevna)]

Micheal,
You can configure token lifetime in /etc/keystone/keystone.conf. Take a look at [token] section parameter expiration. If it works, please close this bug.

[Micheal Jones (micheal-s)]

That's the workaround we are currently using (we set it back to the Havana default of 8 hours) - but that does not fix the issue of the snapshot being uploaded with an expired token. It also makes it impossible to use shorter lived tokens from a security standpoint.

[Micheal Jones (micheal-s)]

Should have read "fix the issue of snapshotting failing because of an expired token" instead of "fix the issue of the snapshot being uploaded with an expired token".

[Nikolay Starodubtsev (starodubcevna)]

My idea is to use trust instead of token for the image upload. Need to make some discovery in this direction, when I'll update the bug description.

[Micheal Jones (micheal-s)]

Much appreciated!

On Sep 9, 2014, at 9:17, Nikolay Starodubtsev <email address hidden> wrote:

> My idea is to use trust instead of token for the image upload. Need to
> make some discovery in this direction, when I'll update the bug
> description.
>
> ** Project changed: keystone => nova
>
> ** Also affects: glance
> Importance: Undecided
> Status: New
>
> ** Changed in: glance
> Assignee: (unassigned) => Nikolay Starodubtsev (starodubcevna)
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1366911
>
> Title:
> Nova does not ensure a valid token is available if snapshot process
> exceeds token lifetime
>
> Status in OpenStack Image Registry and Delivery Service (Glance):
> New
> Status in OpenStack Compute (Nova):
> New
>
> Bug description:
> Recently we encountered the following issue due to the change in
> Icehouse for the default lifetime of a token before it expires. It's
> now 1 hour, while previously it was 8.
>
> If a snapshot process takes longer than an hour, when it goes to the
> next phase it will fail with a 401 Unauthorized error because it has
> an invalid token.
>
> In our specific example the following would take place:
>
> 1. User would set a snapshot to begin and a token would be associated with this request.
> 2. Snapshot would be created, compression time would take about 55 minutes. Enough to just push the snapshotting of this instance over the 60 minute mark.
> 3. Upon Image Upload ("Uploading image data for image" in the logs) Nova would then return a 401 Unauthorized error stating "This server could not verify that you are authorized to access the document you requested. Either you supplied the wrong credentials (e.g., bad password), or your browser does not understand how to supply the credentials required."
>
> Icehouse 2014.1.2, KVM as the hypervisor.
>
> The workaround is to specify a longer token timeout - however limits
> the ability to set short token expirations.
>
> A possible solution may be to get a new/refresh the token if the time
> has exceeded the timeout.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/glance/+bug/1366911/+subscriptions

[Sean Dague (sdague)]

I think the real issue here is that they clients need to revalidate tokens, which they don't, so in this case it's really a glanceclient bug.

[Nikolay Starodubtsev (starodubcevna)]

Thx, Sean! You should be right.

[Micheal Jones (micheal-s)]

sdague: Should this then also be a Horizon bug?

[Ian Cordasco (icordasc)]

This should be addressed with the availability of trusts in keystone. I believe glanceclient supports the usage of trusts now so this should be fixed.