Setting quota fails saying admin project is not a valid project

[Juan Antonio Osorio Robles]

This is what's in the logs http://logs.openstack.org/15/359215/63/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/3465882/console.html#_2017-02-24_11_07_08_893276

2017-02-24 11:07:08.893276 | 2017-02-24 11:07:02.000 | 2017-02-24 11:07:02,929 INFO: + openstack quota set --cores -1 --instances -1 --ram -1 b0fe52b0ac15450ba0a38ac9acd8fea8
2017-02-24 11:07:08.893365 | 2017-02-24 11:07:08.000 | 2017-02-24 11:07:08,674 INFO: Project ID b0fe52b0ac15450ba0a38ac9acd8fea8 is not a valid project. (HTTP 400) (Request-ID: req-9e0a00b7-75ae-41d5-aeed-705bb1a54bae)
2017-02-24 11:07:08.893493 | 2017-02-24 11:07:08.000 | 2017-02-24 11:07:08,758 INFO: [2017-02-24 11:07:08,757] (os-refresh-config) [ERROR] during post-configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/post-configure.d']' returned non-zero exit status 1]

[Juan Antonio Osorio Robles (juan-osorio-robles)]

This is because checking for the validity of the project_id was recently added to nova with this patch https://github.com/openstack/nova/commit/f6fbfc7ff07b790ef052a759552c69429b3d79c7 However, it seems that it's tied to using keystone v3, while it should be using the keystoneclient's functions and attempt to be version agnostic.

[Sean Dague (sdague)]

Keystone v2 has been deprecated since Mitaka, I'm actually kind of confused the Tripleo Pike has not deployed keystone v3.

[Sean Dague (sdague)]

In looking at the logs (to the best that I can)

Tripleo has keystone v3, it's being used correctly by the undercloud.

The project exists in keystone v3.

The call to set the quota is getting a 404 from keystone v3 here in nova - http://logs.openstack.org/15/359215/63/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/3465882/logs/undercloud/var/log/nova/nova-api.txt.gz#_2017-02-24_11_07_08_669

At 11:07:08.669

I can't actually find the corresponding keystone service log there - http://logs.openstack.org/15/359215/63/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/3465882/logs/undercloud/var/log/keystone/keystone.txt.gz.

It's also incredibly hard to see what is going on because the user / project names aren't in the context dump for any of the logs. That should be fixed.

[Sean Dague (sdague)]

Keystone specified that a 404 means the project does not exist, not that you by policy can't see that. We need to get keystone folks engaged on this I think.

[Sean Dague (sdague)]

http://logs.openstack.org/15/359215/63/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/3465882/logs/undercloud/var/log/httpd/keystone_wsgi_main_access.txt.gz

The following request was made:

192.168.24.1 - - [24/Feb/2017:11:07:08 +0000] "GET /v2.0/v3/projects/b0fe52b0ac15450ba0a38ac9acd8fea8 HTTP/1.1" 404 93 "-" "nova-api keystoneauth1/2.18.0 python-requests/2.11.1 CPython/2.7.5"

Is the identity endpoint incorrectly set to http://............./v2.0?

An 'openstack endpoint list' as admin during the failure would be important here.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/438017

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/438049

[Sean Dague (sdague)]

On the Nova side we'll make a log error more explicit so that it's easier to understand what is needed.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/438017
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=6777a24d19ed407a709a424def0919a672d22917
Submitter: Jenkins
Branch: master

commit 6777a24d19ed407a709a424def0919a672d22917
Author: Emilien Macchi <email address hidden>
Date: Fri Feb 24 12:35:15 2017 -0500

    switch keystone endppoints to be versionless

    Switch Keystone endpoints to be versionless so Keystone v3 can be used
    by default on the undercloud.

    Change-Id: I1722972b7231d6d1fe02e1214af71d5296f7da9d
    Related-Bug: #1667679

[Arx Cruz (arxcruz)]

This fix the issue in overcloud, however, undercloud still failing in tempest tests [1]

Shall I open another bug pointing the undercloud?

The output is:

Traceback (most recent call last):
  File "/home/jenkins/tempest/tempest/api/compute/admin/test_quotas.py", line 132, in test_delete_quota
    quota_set_default = (self.adm_client.show_quota_set(project_id)
  File "/home/jenkins/tempest/tempest/lib/services/compute/quotas_client.py", line 31, in show_quota_set
    resp, body = self.get(url)
  File "/home/jenkins/tempest/tempest/lib/common/rest_client.py", line 291, in get
    return self.request('GET', url, extra_headers, headers)
  File "/home/jenkins/tempest/tempest/lib/services/compute/base_compute_client.py", line 48, in request
    method, url, extra_headers, headers, body, chunked)
  File "/home/jenkins/tempest/tempest/lib/common/rest_client.py", line 664, in request
    self._error_checker(resp, resp_body)
  File "/home/jenkins/tempest/tempest/lib/common/rest_client.py", line 766, in _error_checker
    raise exceptions.BadRequest(resp_body, resp=resp)
tempest.lib.exceptions.BadRequest: Bad request
Details: {u'message': u'Project ID 72e6a71f37fd44fea58d9dba99d76936 is not a valid project.', u'code': 400}

And here's the overcloud catalog list for the overcloud:

+------------+----------------+-------------------------------------------------------------------------+
| Name | Type | Endpoints |
+------------+----------------+-------------------------------------------------------------------------+
| placement | placement | regionOne |
| | | publicURL: https://10.0.0.5:13778/placement |
| | | internalURL: http://172.16.2.12:8778/placement |
| | | adminURL: http://172.16.2.12:8778/placement |
| | | |
| neutron | network | regionOne |
| | | publicURL: https://10.0.0.5:13696 |
| | | internalURL: http://172.16.2.12:9696 |
| | | adminURL: http://172.16.2.12:9696 |
| | | |
| cinderv2 | volumev2 | regionOne |
| | | publicURL: https://10.0.0.5:13776/v2/8b544eb8e496437193a5a3d12a9bc427 |
| | | internalURL: |
| | | http://172.16.2.12:8776/v2/8b544eb8e496437193a5a3d12a9bc427 |
| | | adminURL: http://172.16.2.12:8776/v2/8b544eb8e496437193a5a3d12a9bc427 |
...

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/438620

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/438620
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=40a50031f37df0f0cde53e3f3c15ffe407fbdcbd
Submitter: Jenkins
Branch: master

commit 40a50031f37df0f0cde53e3f3c15ffe407fbdcbd
Author: Juan Antonio Osorio Robles <email address hidden>
Date: Mon Feb 27 18:54:45 2017 +0200

    Deploy versionless keystone endpoints (for keystone only)

    The default is to deploy v2.0 endpoints, but this is not the recommended
    approach. we should instead be using versionless endpoints

    Change-Id: Icbfae1c2ff2b7312646fd8e817dd8209220a0d96
    Related-Bug: #1667679

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/448614

[OpenStack Infra (hudson-openstack)]

Change abandoned by Emilien Macchi (<email address hidden>) on branch: stable/ocata
Review: https://review.openstack.org/448614

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/448614
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=803da62a305a31101e5d408af26a243f43fa33ef
Submitter: Jenkins
Branch: stable/ocata

commit 803da62a305a31101e5d408af26a243f43fa33ef
Author: Juan Antonio Osorio Robles <email address hidden>
Date: Mon Feb 27 18:54:45 2017 +0200

    Deploy versionless keystone endpoints (for keystone only)

    The default is to deploy v2.0 endpoints, but this is not the recommended
    approach. we should instead be using versionless endpoints

    Change-Id: Icbfae1c2ff2b7312646fd8e817dd8209220a0d96
    Related-Bug: #1667679
    (cherry picked from commit 40a50031f37df0f0cde53e3f3c15ffe407fbdcbd)

[Mark Goddard (mgoddard)]

I'm also hitting this issue using kolla-ansible on stable/ocata. I'll raise a bug in that project but thought it would be worth commenting here. My question is though, is this a bug in Nova? Are Keystone endpoints with explicit versions supported?

[Daniel Messer (dmesser)]

@mgoddard: I was facing the same issue with kolla-ansible. Manually updating the endpoints corrected the issue for me. My symptom initially was "Unable to retrieve default quota list" until I dug deeper and found the "Project ID <UUID> is not a valid project". The project in question was indeed the admin project as I as logged in as admin at the time to create a new project.

Did you end up filing the bug in Kolla? I don't see it.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/463500

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/463500
Committed: https://git.openstack.org/cgit/openstack/kolla-kubernetes/commit/?id=04c44218662e70911ee66b4ac709def924d3e764
Submitter: Jenkins
Branch: master

commit 04c44218662e70911ee66b4ac709def924d3e764
Author: Chung Chih, Hung <email address hidden>
Date: Tue May 9 17:17:29 2017 +0800

    Keystone endpoint should be versionless

    Nova's quota-set api will validate project id after pike release, but it
    will force to using keystone version v3 with following url
    "/v3/projects/%s". If keystone endpoint suffix with version, then
    request url will be "/v3/v3/projects/%s". Nova will raise bad request
    and quota-set api can not work.

    The keystone endpoint should be versionless.

    Change-Id: I202e212db22af4f56a43283db67b6b583c39a6dd
    Related-Bug: 1667679

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/438049
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4a2009a6fa2017013102d3d43e670334ebc60d9f
Submitter: Jenkins
Branch: master

commit 4a2009a6fa2017013102d3d43e670334ebc60d9f
Author: Sean Dague <email address hidden>
Date: Fri Feb 24 13:56:03 2017 -0500

    Be more tolerant of keystone catalog configuration

    The previous attempt at validating against keystone assumed that users
    had an unversioned endpoint in their catalog. Most do not. This uses
    the keystoneauth discovery magic to do the right thing whether or not
    they have the versioned or unversioned endpoint, as long as they have
    a working v3 installation.

    It is also much more verbose on log messages when things go wrong.

    Change-Id: I509d406ca1f1f7b064aaca88645ad17685827267
    Related-Bug: #1667679
    Closes-Bug: #1693228

[Matt Riedemann (mriedem)]

I'm going to remove ocata from nova for this bug. I'm not sure why anyone would be hitting this in Ocata since this code didn't exist in Nova until Pike.