ERROR in h-eng log after successful tempest run

[David Kranz]

From http://logs.openstack.org/78/45578/11/check/gate-tempest-devstack-vm-full/a991819/logs/screen-h-eng.txt.gz

2013-09-23 14:26:40.352 24996 ERROR heat.openstack.common.rpc.amqp [-] Exception during message handling
2013-09-23 14:26:40.352 24996 TRACE heat.openstack.common.rpc.amqp Traceback (most recent call last):
2013-09-23 14:26:40.352 24996 TRACE heat.openstack.common.rpc.amqp File "/opt/stack/new/heat/heat/openstack/common/rpc/amqp.py", line 461, in _process_data
2013-09-23 14:26:40.352 24996 TRACE heat.openstack.common.rpc.amqp **args)
2013-09-23 14:26:40.352 24996 TRACE heat.openstack.common.rpc.amqp File "/opt/stack/new/heat/heat/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2013-09-23 14:26:40.352 24996 TRACE heat.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2013-09-23 14:26:40.352 24996 TRACE heat.openstack.common.rpc.amqp File "/opt/stack/new/heat/heat/engine/service.py", line 59, in wrapped
2013-09-23 14:26:40.352 24996 TRACE heat.openstack.common.rpc.amqp return func(self, ctx, *args, **kwargs)
2013-09-23 14:26:40.352 24996 TRACE heat.openstack.common.rpc.amqp File "/opt/stack/new/heat/heat/engine/service.py", line 400, in delete_stack
2013-09-23 14:26:40.352 24996 TRACE heat.openstack.common.rpc.amqp st = self._get_stack(cnxt, stack_identity)
2013-09-23 14:26:40.352 24996 TRACE heat.openstack.common.rpc.amqp File "/opt/stack/new/heat/heat/engine/service.py", line 165, in _get_stack
2013-09-23 14:26:40.352 24996 TRACE heat.openstack.common.rpc.amqp raise exception.StackNotFound(stack_name=identity.stack_name)
2013-09-23 14:26:40.352 24996 TRACE heat.openstack.common.rpc.amqp StackNotFound: The Stack (heat791021814) could not be found.
2013-09-23 14:26:40.352 24996 TRACE heat.openstack.common.rpc.amqp
2013-09-23 14:26:40.354 24996 ERROR heat.openstack.common.rpc.common [-] Returning exception The Stack (heat791021814) could not be found. to caller
2013-09-23 14:26:40.354 24996 ERROR heat.openstack.common.rpc.common [-] ['Traceback (most recent call last):\n', ' File "/opt/stack/new/heat/heat/openstack/common/rpc/amqp.py", line 461, in _process_data\n **args)\n', ' File "/opt/stack/new/heat/heat/openstack/common/rpc/dispatcher.py", line 172, in dispatch\n result = getattr(proxyobj, method)(ctxt, **kwargs)\n', ' File "/opt/stack/new/heat/heat/engine/service.py", line 59, in wrapped\n return func(self, ctx, *args, **kwargs)\n', ' File "/opt/stack/new/heat/heat/engine/service.py", line 400, in delete_stack\n st = self._get_stack(cnxt, stack_identity)\n', ' File "/opt/stack/new/heat/heat/engine/service.py", line 165, in _get_stack\n raise exception.StackNotFound(stack_name=identity.stack_name)\n', 'StackNotFound: The Stack (heat791021814) could not be found.\n']

[Angus Salkeld (asalkeld)]

yeah, not sure why we log a massive backtrace just cos' the stack is not there.

[Pablo Andres Fuente (pablo-a-fuente)]

I'm not sure if I'm getting it.
The StackNotFound is an error, so I think is ok to log it as an error. So the thing here is the stack trace for this kind of exceptions, right?
To recap: the idea here is to remove the stack trace of StackNotFound exceptions?

[Clint Byrum (clint-fewbar)]

Excerpts from Pablo Andres Fuente's message of 2013-12-11 17:17:05 UTC:
> I'm not sure if I'm getting it.
> The StackNotFound is an error, so I think is ok to log it as an error. So the thing here is the stack trace for this kind of exceptions, right?
> To recap: the idea here is to remove the stack trace of StackNotFound exceptions?
>

Errors are unexpected conditions.

StackNotFound is not unexpected, people may indeed accidentally try to
delete the same stack twice. That is a DEBUG level message IMO.

[David Kranz (david-kranz)]

Exactly. Operators monitor (or want to) logs for ERROR so they can be alerted when something is potentially wrong with the system. Returning a 404 is only an error as far as the user (client) is concerned, not the operator. This is why we started failing jobs when all the tests passed but new ERRORs appear in the logs.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/62446

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/62446
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=9bb03efbaa0a0e898f92ecb7c7f570ae095f63fe
Submitter: Jenkins
Branch: master

commit 9bb03efbaa0a0e898f92ecb7c7f570ae095f63fe
Author: Pablo Andres Fuente <email address hidden>
Date: Mon Dec 16 15:40:31 2013 -0300

    EventService exception handling to not log errors

    In order to not log a StackNotFound exception as an error, the
    identify_stack() method is now decorated with the
    heat.openstack.common.rpc.client_exceptions wrapper.

    Change-Id: I51adcfe4d86d312c9887e51e6c113f99e0f1b783
    Closes-Bug: #1231135