api.fault notification is never emitted

[Balazs Gibizer]

Description
===========

Setting the notifications.notify_on_api_faults configuration parameter to True and then producing an nova-api fault by injecting a ValueError does not lead to an api.fault notification.

See the nova-api logs here [1] and the patch that inject the ValueError (and some extra traces) here [2].

[1] https://pastebin.com/Eu6rBjNN
[2] https://pastebin.com/en4aFutc

Steps to reproduce
==================
1) apply [2]
2) restart nova services
3) try to create a new aggregate: `nova aggregate-create my-aggregate2`
4) look at the message bus (or the logs if you set the notification driver to log first)

Environment
===========
nova version from master: bc4c91e7be8e4c52506a8f970078ecee6655ee82
config: default devstack +

[notifications]
notify_on_api_faults = True

[oslo_messaging_notifications]
driver = log

[Balazs Gibizer (balazs-gibizer)]

A mail posted to the ML with more implementation details http://lists.openstack.org/pipermail/openstack-dev/2017-June/118639.html

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/505164

[Matt Riedemann (mriedem)]

To be clear, this is just legacy notifications, not versioned notifications.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/505164
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=0d4c3cc65b78bafef69afc45cb156afe38f857c3
Submitter: Jenkins
Branch: master

commit 0d4c3cc65b78bafef69afc45cb156afe38f857c3
Author: Balazs Gibizer <email address hidden>
Date: Mon Sep 18 16:53:32 2017 +0200

    Remove dead code of api.fault notification sending

    Based on the description of the notify_on_api_faults config option [1]
    and the code that uses it [2] nova sends and api.fault notification
    if the nova-api service encounters an unhandle exception.
    There is a FaultWrapper class [3] added to the pipeline of the REST
    request which catches every exception and calls the notification sending.

    Based on some debugging in devstack this FaultWrapper never catches any
    exception. Every REST API method is decorated with expected_errors
    decorator [5] which as a last resort translate the unexpected exception
    to HTTPInternalServerError. In the wsgi stack the actual REST api call is
    guarded with ResourceExceptionHandler context manager [7] which translates
    HTTPException to a Fault [8]. Then Fault is catched and translated to
    the REST response [7]. This way the exception never propagates back to
    the FaultWrapper and therefore the api.fault notification is never emitted.

    Based on the git history of the expected_errors decorator this notification
    was never emitted for v2.1 API and as the v2.0 API now supported with the
    same codebase than v2.1 it is not emitted for v2.0 calls either. As nobody
    reported a bug I assume that nobody tried to use this notification for a very
    long time. Therefore instead of fixing this bug this patch propses to remove
    the dead code.

    See a bit more detailed description on the ML [9].

    [1] https://github.com/openstack/nova/blob/0aeaa2bce8ad15bc2f28e00b30d688514b992e13/nova/conf/notifications.py#L49
    [2] https://github.com/openstack/nova/blob/0aeaa2bce8ad15bc2f28e00b30d688514b992e13/nova/notifications/base.py#L84
    [3] https://github.com/openstack/nova/blob/0aeaa2bce8ad15bc2f28e00b30d688514b992e13/nova/api/openstack/__init__.py#L78
    [5] https://github.com/openstack/nova/blob/0aeaa2bce8ad15bc2f28e00b30d688514b992e13/nova/api/openstack/extensions.py#L325
    [7] https://github.com/openstack/nova/blob/0aeaa2bce8ad15bc2f28e00b30d688514b992e13/nova/api/openstack/wsgi.py#L637
    [8] https://github.com/openstack/nova/blob/0aeaa2bce8ad15bc2f28e00b30d688514b992e13/nova/api/openstack/wsgi.py#L418
    [9] http://lists.openstack.org/pipermail/openstack-dev/2017-June/118639.html

    Change-Id: I608b6ebdc69d31eb2a11ac6479fa4f2e8c20f7d1
    Closes-Bug: #1699115

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.0.0b1 development milestone.