"Unable to get quota information" in horizon when using quantum

[Naveen Joy]

Horizon is unable to retrieve quota information. Please see debug and the local_settings.py files below. However using the nova client, I can see retrieve the quota values.

DEBUG MESSAGE

Recoverable error: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-719d4c08-90d0-416a-9829-86eeaa59b6b3)
Traceback (most recent call last):
  File "/home/njoy/horizon/horizon/usage/base.py", line 110, in quota
    self._quotas = api.nova.tenant_quota_usages(self.request)
  File "/home/njoy/horizon/horizon/utils/memoized.py", line 33, in __call__
    value = self.func(*args)
  File "/home/njoy/horizon/horizon/api/nova.py", line 428, in tenant_quota_usages
    floating_ips = tenant_floating_ip_list(request)
  File "/home/njoy/horizon/horizon/api/nova.py", line 252, in tenant_floating_ip_list
    return novaclient(request).floating_ips.list()
  File "/usr/local/lib/python2.7/dist-packages/python_novaclient-2.6.10-py2.7.egg/novaclient/v1_1/floating_ips.py", line 35, in list
    return self._list("/os-floating-ips", "floating_ips")
  File "/usr/local/lib/python2.7/dist-packages/python_novaclient-2.6.10-py2.7.egg/novaclient/base.py", line 62, in _list
    _resp, body = self.api.client.get(url)
  File "/usr/local/lib/python2.7/dist-packages/python_novaclient-2.6.10-py2.7.egg/novaclient/client.py", line 165, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/python_novaclient-2.6.10-py2.7.egg/novaclient/client.py", line 153, in _cs_request
    **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/python_novaclient-2.6.10-py2.7.egg/novaclient/client.py", line 135, in _time_request
    resp, body = self.request(url, method, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/python_novaclient-2.6.10-py2.7.egg/novaclient/client.py", line 129, in request
    raise exceptions.from_response(resp, body)
ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-719d4c08-90d0-416a-9829-86eeaa59b6b3)
[24/Aug/2012 00:53:49] "GET /nova/ HTTP/1.1" 200 9791
[24/Aug/2012 00:53:49] "GET /i18n/js/horizon/ HTTP/1.1" 200 2158
unable to retrieve service catalog with token
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/keystoneclient/v2_0/client.py", line 132, in _extract_service_catalog
    endpoint_type='adminURL')
  File "/usr/local/lib/python2.7/dist-packages/keystoneclient/service_catalog.py", line 62, in url_for
    raise exceptions.EndpointNotFound('Endpoint not found.')
EndpointNotFound: Endpoint not found.

HORIZON LOCAL SETTINGS FILE:
import os

from django.utils.translation import ugettext_lazy as _

DEBUG = True
TEMPLATE_DEBUG = DEBUG

# Set SSL proxy settings:
# For Django 1.4+ pass this header from the proxy after terminating the SSL,
# and don't forget to strip it from the client's request.
# For more information see:
# https://docs.djangoproject.com/en/1.4/ref/settings/#secure-proxy-ssl-header
# SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTOCOL', 'https')

# Specify a regular expression to validate user passwords.
# HORIZON_CONFIG = {
# "password_validator": {
# "regex": '.*',
# "help_text": _("Your password does not meet the requirements.")
# }
# }

LOCAL_PATH = os.path.dirname(os.path.abspath(__file__))

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'dash',
        'USER': 'root',
        'PASSWORD': '$pass',
        'HOST': '192.168.xxx.yyy',
        'default-character-set': 'utf8'
    },
}

QUANTUM_ENABLED = True
SWIFT_ENABLED = False

# Set custom secret key:
# You can either set it to a specific value or you can let horizion generate a
# default secret key that is unique on this machine, e.i. regardless of the
# amount of Python WSGI workers (if used behind Apache+mod_wsgi): However, there
# may be situations where you would want to set this explicitly, e.g. when
# multiple dashboard instances are distributed on different machines (usually
# behind a load-balancer). Either you have to make sure that a session gets all
# requests routed to the same dashboard instance or you set the same SECRET_KEY
# for all of them.
# from horizon.utils import secret_key
# SECRET_KEY = secret_key.generate_or_read_from_file(os.path.join(LOCAL_PATH, '.secret_key_store'))

# We recommend you use memcached for development; otherwise after every reload
# of the django development server, you will have to login again. To use
# memcached set CACHE_BACKED to something like 'memcached://127.0.0.1:11211/'
CACHE_BACKEND = 'locmem://'

# Send email to the console by default
EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'
# Or send them to /dev/null
#EMAIL_BACKEND = 'django.core.mail.backends.dummy.EmailBackend'

# Configure these for your outgoing email host
# EMAIL_HOST = 'smtp.my-company.com'
# EMAIL_PORT = 25
# EMAIL_HOST_USER = 'djangomail'
# EMAIL_HOST_PASSWORD = 'top-secret!'

# For multiple regions uncomment this configuration, and add (endpoint, title).
# AVAILABLE_REGIONS = [
# ('http://cluster1.example.com:5000/v2.0', 'cluster1'),
# ('http://cluster2.example.com:5000/v2.0', 'cluster2'),
# ]

OPENSTACK_HOST = "192.168.xxx.yyy"
OPENSTACK_KEYSTONE_URL = "http://%s:5000/v2.0" % OPENSTACK_HOST
# OPENSTACK_KEYSTONE_URL = "http://%s:35357/v2.0" % OPENSTACK_HOST
OPENSTACK_KEYSTONE_DEFAULT_ROLE = "Member"
# OPENSTACK_ADMIN_TOKEN="ADMIN"

# The OPENSTACK_KEYSTONE_BACKEND settings can be used to identify the
# capabilities of the auth backend for Keystone.
# If Keystone has been configured to use LDAP as the auth backend then set
# can_edit_user to False and name to 'ldap'.
#
# TODO(tres): Remove these once Keystone has an API to identify auth backend.
OPENSTACK_KEYSTONE_BACKEND = {
    'name': 'native',
    'can_edit_user': True
}

# OPENSTACK_ENDPOINT_TYPE specifies the endpoint type to use for the endpoints
# in the Keystone service catalog. Use this setting when Horizon is running
# external to the OpenStack environment. The default is 'internalURL'.
#OPENSTACK_ENDPOINT_TYPE = "publicURL"

# The number of Swift containers and objects to display on a single page before
# providing a paging element (a "more" link) to paginate results.
API_RESULT_LIMIT = 1000

# The timezone of the server. This should correspond with the timezone
# of your entire OpenStack installation, and hopefully be in UTC.
TIME_ZONE = "UTC"

LOGGING = {
        'version': 1,
        # When set to True this will disable all logging except
        # for loggers specified in this configuration dictionary. Note that
        # if nothing is specified here and disable_existing_loggers is True,
        # django.db.backends will still log unless it is disabled explicitly.
        'disable_existing_loggers': False,
        'handlers': {
            'null': {
                'level': 'DEBUG',
                'class': 'django.utils.log.NullHandler',
                },
            'console': {
                # Set the level to "DEBUG" for verbose output logging.
                'level': 'INFO',
                'class': 'logging.StreamHandler',
                },
            },
        'loggers': {
            # Logging from django.db.backends is VERY verbose, send to null
            # by default.
            'django.db.backends': {
                'handlers': ['null'],
                'propagate': False,
                },
            'horizon': {
                'handlers': ['console'],
                'propagate': False,
            },
            'openstack_dashboard': {
                'handlers': ['console'],
                'propagate': False,
            },
            'novaclient': {
                'handlers': ['console'],
                'propagate': False,
            },
            'keystoneclient': {
                'handlers': ['console'],
                'propagate': False,
            },
            'glanceclient': {
                'handlers': ['console'],
                'propagate': False,
            },
            'nose.plugins.manager': {
                'handlers': ['console'],
                'propagate': False,
            }
        }
}

[Gabriel Hurley (gabriel-hurley)]

The traceback indicates that something went wrong with Nova (hence the 500 error code). I'd have to see the nova logs to even begin debugging this.

It would also be helpful to know how you installed the various OpenStack components and what versions you're running of Horizon, Nova and python-novaclient.

Lastly, it would be helpful to know if everything works *except* this, or if this is just the first error you saw.

[Naveen Joy (najoy)]

Please find the requested information in the attached file. Let me know if you need anything else. Thanks for looking into this issue!.

[Gabriel Hurley (gabriel-hurley)]

Thanks for following up, Naveen. The information in that attachment was exactly what we needed.

The bug you're seeing is way deep down in Nova (and has nothing to do with Horizon or novaclient) so I'm adding Nova to this bug report, but closing it for Horizon. Hopefully they can follow up with you to see if this is a known issue or a new one.

[Mark McLoughlin (markmc)]

From the attachment, the issue appears to be this:

  nova/api/openstack/compute/contrib/floating_ips.py", line 154, in index
     floating_ips = self.network_api.get_floating_ips_by_project(context)
  nova/network/quantumv2/api.py", line 260, in get_floating_ips_by_project
      raise NotImplementedError()

Looks like Horizon is trying to list floating IPs from Nova and this doesn't work with Quantum

Serious issue, but unclear whether this will be fixed in Horizon or Nova at this point - targeting to nova's rc1 milestone for now

[dan wendlandt (danwent)]

Its pretty easy to work around this in Nova, so I can make that change.

The "right" fix is probably in Horizon, as when Quantum is in use, the Quantum API should be queried for floating IPs, not the nova API. The issue is that Horizon does not yet have support for the Quantum L3 extensions, which landed extremely late in Folsom, so its not clear that the "right" fix makes sense in Folsom .

[Mark McLoughlin (markmc)]

Re-opening in Horizon for it to be re-looked at there given Dan's comment

[dan wendlandt (danwent)]

The workaround for this particular error message in Nova is trivial:

diff --git a/nova/network/quantumv2/api.py b/nova/network/quantumv2/api.py
index 990af4c..7db4525 100644
--- a/nova/network/quantumv2/api.py
+++ b/nova/network/quantumv2/api.py
@@ -254,7 +254,7 @@ class API(base.Base):
         raise NotImplementedError()

     def get_floating_ips_by_project(self, context):
- raise NotImplementedError()
+ return []

     def get_floating_ips_by_fixed_address(self, context, fixed_address):
         raise NotImplementedError()

However, after thinking about this more, I think its really that we either need to:

1) actually proxy Nova Floating IP calls to Quantum
2) prevent Horizon from calling Nova floating IP calls (here and elsewhere) when Quantum is enabled.

The reason is that it doesn't seem right for Horizon to suggest that there are working floating IPs when there are not.

Our original plan was to do #1 as part of F-3 but this bug wasn't finished then and working on it as an FFE stalled and the current author seems to not actively be doing anything (https://bugs.launchpad.net/quantum/+bug/1031119).

Akihiro has volunteered to do this work. I think it would be great for him to do this, and I think it would be pretty straightforward. That said, this is a change in Nova and is definitely non-trivial, so its not clear that it should be added for Folsom. That call is up to the Nova folks. It can definitely be put in Grizzly though.

[dan wendlandt (danwent)]

Hi folks, just wanted to ping again about this. Are we planning a change in Horizon or Nova?

[Gabriel Hurley (gabriel-hurley)]

For what it's worth I'd rather not have this be a "we can't fix it so let's add another setting in Horizon" scenario... Patches to Nova/Quantum preferred ;-)

[Thierry Carrez (ttx)]

Vish says "I would lean towards just returning [] for now and get the proxying done asap in Grizzly."

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/12820

[Gabriel Hurley (gabriel-hurley)]

I'm good with that fix in Nova for Folsom. Since this is only being used for an informational quota display it's not mission-critical that it be correct with Quantum right away. It can just be a known limitation.

Closing for Horizon now that we have a proposed solution.

[dan wendlandt (danwent)]

pushed Nova work around: https://review.openstack.org/12820

Note, this also returns an empty list for the floating ip pools, which
ideally would prevent users from even being able to try and create a
floating IP (which also will not work). This doesn't work perfectly
though, as it seems like Horizon will let a user press the "Allocate
IP" button even if no floating ip pools are available. Horizon
correctly reports an error though, so the issue is minor.

Dan

--
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dan Wendlandt
Nicira, Inc: www.nicira.com
twitter: danwendlandt
~~~~~~~~~~~~~~~~~~~~~~~~~~~

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/12820
Committed: http://github.com/openstack/nova/commit/c0709bdd82c83e16cab6ed854d2ef873eb775473
Submitter: Jenkins
Branch: master

commit c0709bdd82c83e16cab6ed854d2ef873eb775473
Author: Dan Wendlandt <email address hidden>
Date: Tue Sep 11 12:22:23 2012 -0700

    workaround lack of quantum/nova floatingip integration

    bug 1040956

    The Nova/Quantum integration in Folsom does not yet proxy Nova
    floating IP calls to Quantum. However, Horizon automatically
    makes calls to Nova to list floating IPs, which prior to this
    patch resulted in a Horizon error message due to the fact that
    the Quantum integration code in Nova would raise an Unimplemented
    error. This patch changes that behavior to instead return an
    empty list.

    This is a workaround for Folsom. The correct long-term behavior
    is either to:
    1) have Nova proxy all floating-ip calls to Quantum when quantum
    is enabled.
    2) have Horizon use the Quantum floating-ip API when Quantum
    is enabled.

    Change-Id: I5414d935e872a9da108511b50e70545d82b73347