KeyError: 'used' security_group_rule quota missing 'used' key

[Corey Bryant]

On rocky rc1, after attempting to log in to the dashboard I hit:

Internal Server Error: /horizon/project/
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  ...
  File "/usr/lib/python2.7/dist-packages/openstack_dashboard/usage/views.py", line 163, in _process_chart_section
    used = self.usage.limits[key]['used']
    KeyError: 'used'

Full traceback: https://paste.ubuntu.com/p/RcMCjWs8HG/

From openstack_dashboard/usage/views.py:

    def _process_chart_section(self, chart_defs):
        charts = []
        for t in chart_defs:
            if t.quota_key not in self.usage.limits:
                continue
            key = t.quota_key
            used = self.usage.limits[key]['used'] # <--- KeyError
            quota = self.usage.limits[key]['quota']

Further debugging shows we're failing on key='security_group_rule'

chart_def=ChartDef(quota_key='security_group_rule', label=u'Security Group Rules', used_phrase=None, filters=None)
self.usage.limits[key]={'quota': 100}

Notice there's no 'used' key in self.usage.limits. Compare that vs 'security_group' which has:

chart_def=ChartDef(quota_key='security_group', label=u'Security Groups', used_phrase=None, filters=None)
self.usage.limits[key]={'available': 9, 'used': 1, 'quota': 10}

From openstack_dashboard/usage/quotas.py:

    def tally(self, name, value):
        """Adds to the "used" metric for the given quota."""
        value = value or 0 # Protection against None.
        # Start at 0 if this is the first value.
        if 'used' not in self.usages[name]:
            self.usages[name]['used'] = 0

I haven't confirmed but it seems that tally does the initialization of the 'used' key and for some reason that's not happening.

[Corey Bryant (corey.bryant)]

This jumped out at me from _get_tenant_network_usages_legacy() in openstack_dashboard/usage/quotas.py:

# TODO(amotoki): Add security_group_rule?

So I checked and in _get_tenant_network_usages() the following check evaluates to False, causing _get_tenant_network_usages_legacy() to be called:

if neutron.is_extension_supported(request, 'quota_details'):

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/595855

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/595857

[Corey Bryant (corey.bryant)]

Some side notes about why we're using the legacy path..

I'm not quite sure why the quota_details extension is being detected as unsupported. Does it need to be explicitly configured?

neutron-server.log seems to show the 'quota_details' extension getting loaded successfully: https://paste.ubuntu.com/p/TJnhQcZVcf/

However 'openstack extension list --network' isn't showing 'quota_details': https://paste.ubuntu.com/p/WnXR4fcsVd/

[Akihiro Motoki (amotoki)]

It seems you use some non-default configuration for quotas. quota_details extension is enabled by default.
According to the code, the extension will be disabled if some other quota driver other than neutron.db.quota.driver.DbQuotaDriver is configured. The other possible quota driver is neutron.quota.ConfDriver but this driver has been deprecated from Liberty. What configuration do you use?
But this is a bug tracker for horizon, so I will stop this discussion here.

[1] https://github.com/openstack/neutron/blob/master/neutron/extensions/quotasv2_detail.py#L63

[Akihiro Motoki (amotoki)]

Confirmed this is a valid bug.

On the other hand, perhaps we should drop the legacy quota handling in the Stein release because the corresponding neutron quota detail feature was added in Pike release.

[Akihiro Motoki (amotoki)]

The priority is changed to medium because this only happens when the deprecated neutron feature is enabled.

[Corey Bryant (corey.bryant)]

Thanks for taking a look. We have the following config on our neutron API node, so I'm not sure what is going on. I'll need to dig some more.

[quotas]
quota_driver = neutron.db.quota_db.DbQuotaDriver
quota_items = network,subnet,port,security_group,security_group_rule
quota_security_group = 10
quota_security_group_rule = 100
quota_network = 10
quota_subnet = 10
quota_port = 50
quota_vip = 10
quota_pool = 10
quota_member = -1
quota_health_monitors = -1
quota_router = 10
quota_floatingip = 50

[Corey Bryant (corey.bryant)]

Aha! neutron.db.quota.driver.DbQuotaDriver != neutron.db.quota_db.DbQuotaDriver

That was the problem. At least the 'openstack extension list --network' is now returning quota_details.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/596710

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/596710
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-api/commit/?id=109005ad369bd85221ffb0ca8df998d9a25218c3
Submitter: Zuul
Branch: master

commit 109005ad369bd85221ffb0ca8df998d9a25218c3
Author: Corey Bryant <email address hidden>
Date: Mon Aug 27 12:55:24 2018 +0000

    Add neutron.conf for Rocky with updated quota_driver

    Updates the path to the quota_driver as of Rocky.

    Also enable gate-basic-*-rocky amulet tests.

    Change-Id: Iaaeac7e1e8aded85e9cdb2f9fec3344c2fae4ed2
    Closes-Bug: #1788631

[OpenStack Infra (hudson-openstack)]

Change abandoned by Corey Bryant (<email address hidden>) on branch: stable/rocky
Review: https://review.openstack.org/595857
Reason: This patch isn't complete and there's consideration of dropping the legacy quota handling in stein: https://bugs.launchpad.net/horizon/+bug/1788631/comments/6

[OpenStack Infra (hudson-openstack)]

Change abandoned by Corey Bryant (<email address hidden>) on branch: master
Review: https://review.openstack.org/595855
Reason: This patch isn't complete and there's consideration of dropping the legacy quota handling in stein: https://bugs.launchpad.net/horizon/+bug/1788631/comments/6

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/18.08
Review: https://review.openstack.org/613609

[OpenStack Infra (hudson-openstack)]

Change abandoned by Edward Hope-Morley (<email address hidden>) on branch: stable/18.08
Review: https://review.openstack.org/613609
Reason: submitted to wrong branch. will resubmit to master.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/613610

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/613610
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-api/commit/?id=31a7dd32d76394210e53ca36d3fc1fbd210ee368
Submitter: Zuul
Branch: master

commit 31a7dd32d76394210e53ca36d3fc1fbd210ee368
Author: Edward Hope-Morley <email address hidden>
Date: Fri Oct 26 16:08:56 2018 +0100

    Make Queens and Rocky share same conf template

    The original patch for this bug fixed the problem for
    rocky only but queens also suffers from the same
    problem. Since the only diff between the P and R
    templates is this change, move it into Q so that
    Q and R will both get the fix.

    Change-Id: Ib4573aa925cb1b3b9e1fa04e8a3c5bf611223c25
    Related-Bug: #1788631

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/18.08
Review: https://review.openstack.org/613982

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/613982
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-api/commit/?id=ce5973a76a0e7091f6d5d5fdc48dd81b69bc2046
Submitter: Zuul
Branch: stable/18.08

commit ce5973a76a0e7091f6d5d5fdc48dd81b69bc2046
Author: Edward Hope-Morley <email address hidden>
Date: Fri Oct 26 16:08:56 2018 +0100

    Make Queens and Rocky share same conf template

    The original patch for this bug fixed the problem for
    rocky only but queens also suffers from the same
    problem. Since the only diff between the P and R
    templates is this change, move it into Q so that
    Q and R will both get the fix.

    Change-Id: Ib4573aa925cb1b3b9e1fa04e8a3c5bf611223c25
    Related-Bug: #1788631
    (cherry picked from commit 31a7dd32d76394210e53ca36d3fc1fbd210ee368)

[Akihiro Motoki (amotoki)]

Mark this as Won't Fix in horizon considering this only happens when the deprecated neutron quota driver is used.