ERROR: Unable to retrieve volume limit information in horizon

[Anastasia Palkina]

1. Install plugins

https://github.com/openstack/fuel-plugin-detach-keystone
https://github.com/openstack/fuel-plugin-detach-database

2. Create new environment by default
3. Enable both plugins on "Settings" tab
4. Add 1 controller, 1 compute, 1 keystone+database
5. Start deployment. It was successful

6. Login to Horizon
7. When you open some segments, error appears: Unable to retrieve volume limit information. (see screens)
8. There is also error in nova-compute.log on compute (node-1):

2015-12-01 14:12:05 ERR No compute node record for host node-1.domain.tld

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  openstack_version: "2015.1.0-8.0"
  api: "1.0"
  build_number: "233"
  build_id: "233"
  fuel-nailgun_sha: "0a2f99530cf7246acf85db643032a0550168aac5"
  python-fuelclient_sha: "b1ffe1cae9ce7b612d3f746c8e2e2fde6f732748"
  fuel-agent_sha: "bd67efbadabfd8242c979c50b7d61a251621621a"
  fuel-nailgun-agent_sha: "a33a58d378c117c0f509b0e7badc6f0910364154"
  astute_sha: "b60624ee2c5f1d6d805619b6c27965a973508da1"
  fuel-library_sha: "ed9c5c91eda71bfcdcd3e08f91fa3627ea87829c"
  fuel-ostf_sha: "a98973482f839554d90cc1c071d625a01e018cfe"
  fuel-createmirror_sha: "3cb98030d4a12992ea1cda1f464f035980569d2f"
  fuelmenu_sha: "fcb15df4fd1a790b17dd78cf675c11c279040941"
  shotgun_sha: "25a0cc461a9fa4f7684f04cef0ff4ad9aa99a64d"
  network-checker_sha: "0b1b94a9685c6471d6911dff7ecac10b7bd2625f"
  fuel-upgrade_sha: "1e894e26d4e1423a9b0d66abd6a79505f4175ff6"
  fuelmain_sha: "2eca6adc33f02e02cd812e1d4be7c70e05fd07db"

[Matthew Mosesohn (raytrac3r)]

I can't reproduce the compute node part, but the cinder volume part for sure:

2015-12-03 12:08:49.779 24851 INFO cinder.api.openstack.wsgi [req-9df2bb35-c839-4db4-b5bd-2b2c2a6e90f3 0fd5044264c14b4ca7e6b043a9dabc30 05ad882b93044a75aacacc059cb2bf37 - - -] GET http://public.fuel.local:8776/v1/05ad882b93044a75aacacc05
9cb2bf37/os-quota-sets/admin?usage=False
2015-12-03 12:08:49.812 24851 ERROR cinder.api.middleware.fault [req-9df2bb35-c839-4db4-b5bd-2b2c2a6e90f3 0fd5044264c14b4ca7e6b043a9dabc30 05ad882b93044a75aacacc059cb2bf37 - - -] Caught error: Authorization failed: Not Found (HTTP 404) (
HTTP 404)
2015-12-03 12:08:49.824 24851 INFO cinder.api.middleware.fault [req-9df2bb35-c839-4db4-b5bd-2b2c2a6e90f3 0fd5044264c14b4ca7e6b043a9dabc30 05ad882b93044a75aacacc059cb2bf37 - - -] http://public.fuel.local:8776/v1/05ad882b93044a75aacacc059c
b2bf37/os-quota-sets/admin?usage=False returned with HTTP 500
2015-12-03 12:08:49.837 24851 INFO eventlet.wsgi.server [req-9df2bb35-c839-4db4-b5bd-2b2c2a6e90f3 0fd5044264c14b4ca7e6b043a9dabc30 05ad882b93044a75aacacc059cb2bf37 - - -] 240.0.0.2 - - [03/Dec/2015 12:08:49] "GET /v1/05ad882b93044a75aaca
cc059cb2bf37/os-quota-sets/admin?usage=False HTTP/1.1" 500 420 0.359449

[Matthew Mosesohn (raytrac3r)]

I managed to get more debug info:
http://paste.openstack.org/show/wfVJfKUzeRGaMmUZCoKv/

The most noteworthy item is "Making authentication request to http://localhost:5000/v3/auth/tokens get_auth_ref". It shouldn't refer to localhost:5000 for any reason and there are no configs in cinder.conf like this.

[Matthew Mosesohn (raytrac3r)]

Oslo team says that this error doesn't explicitly print its traceback:
https://github.com/openstack/cinder/blob/master/cinder/api/middleware/fault.py#L71

I need MOS-Cinder team to keep looking into this.

[Matthew Mosesohn (raytrac3r)]

https://review.openstack.org/#/c/251937/ fixes this bug for Mitaka (but is unmerged). It would still have to get backported to liberty. I think it still makes sense to fix this in puppet for now.

[Matthew Mosesohn (raytrac3r)]

https://review.openstack.org/#/c/253480/ under review in puppet-cinder

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: mos-8.0
Change author: Matthew Mosesohn <email address hidden>
Review: https://review.fuel-infra.org/14476

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/14476
Submitter: Ivan Berezovskiy <email address hidden>
Branch: mos-8.0

Commit: e3471bba93149bb9b43cf19ef28ac1c1d4ce9561
Author: Matthew Mosesohn <email address hidden>
Date: Mon Dec 7 16:40:36 2015

Add keymgr/encryption_auth_url parameter to cinder API

This value defaults to http://localhost:5000/v3, which works for many
deployments, but if Keystone is located on a separate node,
cinder quota-show $PROJECT_ID fails to contact keystone.

More info about this bug at LP #1516085
Partial-Bug: #1521640

Change-Id: I3457e60e8ce5f2df0830b45c3d18caa88e40f040
(cherry picked from commit 277232eba38ec493bb8d97ad49b2e86457f81fdb)

[Timur Nurlygayanov (tnurlygayanov)]

It is merged to MOS 8.0, status changed to Fix Committed.

[Timur Nurlygayanov (tnurlygayanov)]

Looks like the issue was not fixed, reproduced on MOS 8.0 #307 in tempest run:
https://bugs.launchpad.net/mos/+bug/1527613

[Matthew Mosesohn (raytrac3r)]

https://review.openstack.org/255735 fixes this, but it depends on https://review.openstack.org/255982 . Waiting on reviews still

[Timur Nurlygayanov (tnurlygayanov)]

Ok, looks like we need to wait for new commits... this bug is the reason of 18 failed Tempest tests...

[Timur Nurlygayanov (tnurlygayanov)]

Marked as "blocker-for-qa" - we can't boot VMs because of this issue.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/255735
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=08d935ee860be394f97c99a04836908204e544cf
Submitter: Jenkins
Branch: master

commit 08d935ee860be394f97c99a04836908204e544cf
Author: Matthew Mosesohn <email address hidden>
Date: Tue Dec 8 20:06:35 2015 +0300

    Configure keymgr auth url if identity_uri is used

    Cinder requires a keystone URI for performing quota operations.
    This behiavor is present in Liberty release, but will be fixed
    in Mitaka release to re-use the identity_uri parameter.

    This change adds the parameter keymgr_encryption_auth_url to
    identity_uri + /v3 if identity_uri is set when calling cinder:api.

    Updates openstack-cinder ref to mos-8.0 to pull in
    https://review.fuel-infra.org/#/c/14476/

    This can be reverted after LP#1516085 is in use for Mitaka cycle.

    Closes-Bug: #1521640
    Change-Id: Ide9b35761b25be155d348bf8c8704efc23634482

[Timur Nurlygayanov (tnurlygayanov)]

This issue was successfully fixed and verified.