enabling debug mode in horizon causes 500 erorrs

Bug #1330825 reported by Matthew Mosesohn
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
Medium
Bogdan Dobrelya
4.1.x
Fix Released
Medium
Bogdan Dobrelya
5.0.x
Fix Released
Medium
Bogdan Dobrelya

Bug Description

Deployment:
Ubuntu 12.04 on Fuel 4.1
Neutron with vlan segmentation
3 controllers, 3 storage (Ceph), 20 computes

A user will encounter intermittent 500 Internal Server Error messages. This is usually associated with trying to launch an instance from a snapshot. Repeated attempts eventually work.

Traceback:
2014-06-17T05:29:04.736950+00:00 err: ERROR Not Found: Instance could not be found (HTTP 404) (Request-ID: req-9f25178b-c483-4ad5-9236-fd8ac7c4dcae)
Traceback (most recent call last):
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/project/volumes/tables.py", line 120, in get_attachment_name
    server = api.nova.server_get(request, server_id)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 489, in server_get
    return Server(novaclient(request).servers.get(instance_id), request)
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/servers.py", line 370, in get
    return self._get("/servers/%s" % base.getid(server), "server")
  File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 140, in _get
    _resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 229, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 213, in _cs_request
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 195, in _time_request
    resp, body = self.request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 189, in request
    raise exceptions.from_response(resp, body, url, method)
NotFound: Instance could not be found (HTTP 404) (Request-ID: req-9f25178b-c483-4ad5-9236-fd8ac7c4dcae)

It is only reproducable under high load. Disabling debug mode for Horizon will cause the issue to go away. Perhaps one solution is tuning down the level of debugging to only the services we care about most.

description: updated
description: updated
Changed in fuel:
status: New → Triaged
assignee: Fuel Library Team (fuel-library) → Bogdan Dobrelya (bogdando)
Changed in fuel:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/100887

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/100887
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=db04d3385d0a379ee9dbc5583ade6a5820c5a3ad
Submitter: Jenkins
Branch: master

commit db04d3385d0a379ee9dbc5583ade6a5820c5a3ad
Author: Bogdan Dobrelya <email address hidden>
Date: Wed Jun 18 15:48:05 2014 +0300

    Sync loggers and handlers from Horizon upstream

    In order to prevent too verbose logging, use recommended
    patterns from upstream examples
    https://github.com/openstack/horizon/blob/2014.1.1/openstack_dashboard/local/local_settings.py.example

    Closes-bug: #1330825

    Change-Id: Ie93816388d2bb97d6543c3452398ec15e5c8fd31
    Signed-off-by: Bogdan Dobrelya <email address hidden>

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/5.0)

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/102971

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/5.0)

Reviewed: https://review.openstack.org/102971
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=c9d96153c7d33893ccb01fd1a706f4e13d20c906
Submitter: Jenkins
Branch: stable/5.0

commit c9d96153c7d33893ccb01fd1a706f4e13d20c906
Author: Bogdan Dobrelya <email address hidden>
Date: Wed Jun 18 15:48:05 2014 +0300

    Sync loggers and handlers from Horizon upstream

    In order to prevent too verbose logging, use recommended
    patterns from upstream examples
    https://github.com/openstack/horizon/blob/2014.1.1/openstack_dashboard/local/local_settings.py.example

    Closes-bug: #1330825

    Change-Id: Ie93816388d2bb97d6543c3452398ec15e5c8fd31
    Signed-off-by: Bogdan Dobrelya <email address hidden>

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/4.1)

Fix proposed to branch: stable/4.1
Review: https://review.openstack.org/103074

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/4.1)

Reviewed: https://review.openstack.org/103074
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=df0fefb16cc5f578193b36363c63c7d312303436
Submitter: Jenkins
Branch: stable/4.1

commit df0fefb16cc5f578193b36363c63c7d312303436
Author: Bogdan Dobrelya <email address hidden>
Date: Wed Jun 18 15:48:05 2014 +0300

    Sync loggers and handlers from Horizon upstream

    In order to prevent too verbose logging, use recommended
    patterns from upstream examples
    https://github.com/openstack/horizon/blob/2014.1.1/openstack_dashboard/local/local_settings.py.example

    Closes-bug: #1330825

    Change-Id: Ie93816388d2bb97d6543c3452398ec15e5c8fd31
    Signed-off-by: Bogdan Dobrelya <email address hidden>

tags: added: in-progress
Revision history for this message
Dennis Dmitriev (ddmitriev) wrote :

{"build_id": "2014-09-17_21-40-34", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "11", "auth_required": true, "api": "1.0", "nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d", "production": "docker", "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd", "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13", "feature_groups": ["mirantis"], "release": "5.1", "release_versions": {"2014.1.1-5.1": {"VERSION": {"build_id": "2014-09-17_21-40-34", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "11", "api": "1.0", "nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d", "production": "docker", "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd", "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13", "feature_groups": ["mirantis"], "release": "5.1", "fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"}}}, "fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"}

Changed in fuel:
status: Fix Committed → Fix Released
Revision history for this message
Dennis Dmitriev (ddmitriev) wrote :

{"build_id": "2014-08-11_00-41-17", "mirantis": "yes", "build_number": "422", "nailgun_sha": "8cc1dff8ab43f50bd28a9c081d7d06fbd3831b98", "ostf_sha": "f4f15b4d98459650c1945b0efc30290a619be824", "fuelmain_sha": "fa218a36d2686de6bb36ef8c6b33526c9d802e34", "astute_sha": "55df06b2e84fa5d71a1cc0e78dbccab5db29d968", "release": "4.1.1", "fuellib_sha": "469ed82eae57bd85939678054dfa0260e8dbf895"}

tags: removed: in-progress
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.