enabling debug mode in horizon causes 500 erorrs

Bug #1330825 reported by Matthew Mosesohn on 2014-06-17
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Medium
Bogdan Dobrelya
4.1.x
Medium
Bogdan Dobrelya
5.0.x
Medium
Bogdan Dobrelya

Bug Description

Deployment:
Ubuntu 12.04 on Fuel 4.1
Neutron with vlan segmentation
3 controllers, 3 storage (Ceph), 20 computes

A user will encounter intermittent 500 Internal Server Error messages. This is usually associated with trying to launch an instance from a snapshot. Repeated attempts eventually work.

Traceback:
2014-06-17T05:29:04.736950+00:00 err: ERROR Not Found: Instance could not be found (HTTP 404) (Request-ID: req-9f25178b-c483-4ad5-9236-fd8ac7c4dcae)
Traceback (most recent call last):
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/project/volumes/tables.py", line 120, in get_attachment_name
    server = api.nova.server_get(request, server_id)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 489, in server_get
    return Server(novaclient(request).servers.get(instance_id), request)
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/servers.py", line 370, in get
    return self._get("/servers/%s" % base.getid(server), "server")
  File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 140, in _get
    _resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 229, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 213, in _cs_request
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 195, in _time_request
    resp, body = self.request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 189, in request
    raise exceptions.from_response(resp, body, url, method)
NotFound: Instance could not be found (HTTP 404) (Request-ID: req-9f25178b-c483-4ad5-9236-fd8ac7c4dcae)

It is only reproducable under high load. Disabling debug mode for Horizon will cause the issue to go away. Perhaps one solution is tuning down the level of debugging to only the services we care about most.

description: updated
description: updated
Changed in fuel:
status: New → Triaged
assignee: Fuel Library Team (fuel-library) → Bogdan Dobrelya (bogdando)
Changed in fuel:
status: Triaged → In Progress

Reviewed: https://review.openstack.org/100887
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=db04d3385d0a379ee9dbc5583ade6a5820c5a3ad
Submitter: Jenkins
Branch: master

commit db04d3385d0a379ee9dbc5583ade6a5820c5a3ad
Author: Bogdan Dobrelya <email address hidden>
Date: Wed Jun 18 15:48:05 2014 +0300

    Sync loggers and handlers from Horizon upstream

    In order to prevent too verbose logging, use recommended
    patterns from upstream examples
    https://github.com/openstack/horizon/blob/2014.1.1/openstack_dashboard/local/local_settings.py.example

    Closes-bug: #1330825

    Change-Id: Ie93816388d2bb97d6543c3452398ec15e5c8fd31
    Signed-off-by: Bogdan Dobrelya <email address hidden>

Changed in fuel:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/102971
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=c9d96153c7d33893ccb01fd1a706f4e13d20c906
Submitter: Jenkins
Branch: stable/5.0

commit c9d96153c7d33893ccb01fd1a706f4e13d20c906
Author: Bogdan Dobrelya <email address hidden>
Date: Wed Jun 18 15:48:05 2014 +0300

    Sync loggers and handlers from Horizon upstream

    In order to prevent too verbose logging, use recommended
    patterns from upstream examples
    https://github.com/openstack/horizon/blob/2014.1.1/openstack_dashboard/local/local_settings.py.example

    Closes-bug: #1330825

    Change-Id: Ie93816388d2bb97d6543c3452398ec15e5c8fd31
    Signed-off-by: Bogdan Dobrelya <email address hidden>

Reviewed: https://review.openstack.org/103074
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=df0fefb16cc5f578193b36363c63c7d312303436
Submitter: Jenkins
Branch: stable/4.1

commit df0fefb16cc5f578193b36363c63c7d312303436
Author: Bogdan Dobrelya <email address hidden>
Date: Wed Jun 18 15:48:05 2014 +0300

    Sync loggers and handlers from Horizon upstream

    In order to prevent too verbose logging, use recommended
    patterns from upstream examples
    https://github.com/openstack/horizon/blob/2014.1.1/openstack_dashboard/local/local_settings.py.example

    Closes-bug: #1330825

    Change-Id: Ie93816388d2bb97d6543c3452398ec15e5c8fd31
    Signed-off-by: Bogdan Dobrelya <email address hidden>

tags: added: in-progress
Dennis Dmitriev (ddmitriev) wrote :

{"build_id": "2014-09-17_21-40-34", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "11", "auth_required": true, "api": "1.0", "nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d", "production": "docker", "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd", "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13", "feature_groups": ["mirantis"], "release": "5.1", "release_versions": {"2014.1.1-5.1": {"VERSION": {"build_id": "2014-09-17_21-40-34", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "11", "api": "1.0", "nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d", "production": "docker", "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd", "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13", "feature_groups": ["mirantis"], "release": "5.1", "fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"}}}, "fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"}

Changed in fuel:
status: Fix Committed → Fix Released
Dennis Dmitriev (ddmitriev) wrote :

{"build_id": "2014-08-11_00-41-17", "mirantis": "yes", "build_number": "422", "nailgun_sha": "8cc1dff8ab43f50bd28a9c081d7d06fbd3831b98", "ostf_sha": "f4f15b4d98459650c1945b0efc30290a619be824", "fuelmain_sha": "fa218a36d2686de6bb36ef8c6b33526c9d802e34", "astute_sha": "55df06b2e84fa5d71a1cc0e78dbccab5db29d968", "release": "4.1.1", "fuellib_sha": "469ed82eae57bd85939678054dfa0260e8dbf895"}

tags: removed: in-progress
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers