Glance writes stack trace to logs in correct situations

Bug #1699512 reported by Kairat Kushaev
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Fix Released
Medium
Kairat Kushaev
9.x
Fix Released
Medium
Kairat Kushaev

Bug Description

When missing images are requested with glance API v1, long python-style
traceback is stored in glance-api log file. This affects many monitoring
systems and complicates the investigation of real techical issues.
we need get rid of this long stack trace and show error message only.
This affects only glance with registry backend- so it is not applicable to glance v2 and affects v1 only.
Test Case:
1.install openstack cloud, enable v1 registry for glance
2. glance --image-api-version 1 image-show not_existing
AR:a long stack trace with NotFound exception in /var/log/glance/api.log
ER: a short message with 404 exception in /var/log/glance.api.log

Revision history for this message
Kairat Kushaev (kkushaev) wrote :

https://bugs.launchpad.net/glance/+bug/1533970
https://bugs.launchpad.net/glance/+bug/1654270- upstream bugs. upstream fix were rejected cause v1 is not supported in upstream

Changed in mos:
milestone: none → 8.0-mu-5
importance: Undecided → Medium
assignee: nobody → Kairat Kushaev (kkushaev)
Revision history for this message
Kairat Kushaev (kkushaev) wrote :
Revision history for this message
Danil Akhmetov (dinobot) wrote :

Important notice: this patch also requires another change which happened in Newton.

[1] https://bugs.launchpad.net/glance/+bug/1450209
[2] https://review.openstack.org/#/c/355960/

That patch is a workaround, and we still should rise the proper exception there.

Originally I faced this issue when StackLight started generating hundreds of CRIT alerts for Glance. To my surprise, I found Glance fully operational: I investigated this issue and proposed the patch quick.

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on openstack/glance (openstack-ci/fuel-8.0/liberty)

Change abandoned by Kairat Kushaev <email address hidden> on branch: openstack-ci/fuel-8.0/liberty
Review: https://review.fuel-infra.org/35695
Reason: Seems thathttps://review.fuel-infra.org/#/c/35730/ is enough to fix the issue

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on openstack/glance (9.0/mitaka)

Change abandoned by Kairat Kushaev <email address hidden> on branch: 9.0/mitaka
Review: https://review.fuel-infra.org/35693
Reason: Seems thathttps://review.fuel-infra.org/#/c/35730/ is enough to fix the issue

Revision history for this message
Kairat Kushaev (kkushaev) wrote :

I proposed patches that actually fix the issues:
 https://review.fuel-infra.org/#/c/35730/ - 9.0
tps://review.fuel-infra.org/#/c/35731/ - 8.0

Revision history for this message
Kairat Kushaev (kkushaev) wrote :
tags: added: customer-found
description: updated
description: updated
Changed in mos:
status: New → In Progress
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/glance (openstack-ci/fuel-8.0/liberty)

Reviewed: https://review.fuel-infra.org/35731
Submitter: Pkgs Jenkins <email address hidden>
Branch: openstack-ci/fuel-8.0/liberty

Commit: 3eb8ee7adce08d64605d5ce799b8c14704a95c9b
Author: Sean Dague <email address hidden>
Date: Thu Jun 22 12:11:39 2017

Stop stack tracing on 404s

Glance has had this persistent bit of log noise where trying to GET a
non existent image creates a stack trace at ERROR log level. This
makes glance api logs extra noisy, and confuses when actual errors
happen.
(cherry-picked from commit 5caf1c739e190338e87be8bcd880cb88b0920299)
Conflicts:
       glance/registry/client/v1/client.py
Closes-Bug:#1699512
Change-Id: Ic08f3a3885f31433f2416e317b412e743512f797

Changed in mos:
status: In Progress → Fix Committed
Revision history for this message
Dmitry (dtsapikov) wrote :

Verified on 8.0+mu5

tags: added: on-verification
Changed in mos:
status: Fix Committed → Fix Released
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/glance (9.0/mitaka)

Reviewed: https://review.fuel-infra.org/35730
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: 8bb07dcc8ac414ed5f59e6f134f67c61abf6eb74
Author: Sean Dague <email address hidden>
Date: Thu Jun 22 09:05:36 2017

Stop stack tracing on 404s

Glance has had this persistent bit of log noise where trying to GET a
non existent image creates a stack trace at ERROR log level. This
makes glance api logs extra noisy, and confuses when actual errors
happen.
(cherry-picked from commit 5caf1c739e190338e87be8bcd880cb88b0920299)
Closes-Bug: #1699512
Change-Id: Ic08f3a3885f31433f2416e317b412e743512f797

Revision history for this message
Ilya Bumarskov (ibumarskov) wrote :

Can't reproduce bug on Fuel 9.2 MU-2. We get HTTP 200 on request "glance --os-image-api-version 1 image-show not_existing" instead 404 error. However bug also isn't reproducible on MU-3 (snapshots/9.0-2017-10-09-094324).

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.