api.fault notification is never emitted

Bug #1699115 reported by Balazs Gibizer on 2017-06-20
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Balazs Gibizer

Bug Description


Setting the notifications.notify_on_api_faults configuration parameter to True and then producing an nova-api fault by injecting a ValueError does not lead to an api.fault notification.

See the nova-api logs here [1] and the patch that inject the ValueError (and some extra traces) here [2].

[1] https://pastebin.com/Eu6rBjNN
[2] https://pastebin.com/en4aFutc

Steps to reproduce
1) apply [2]
2) restart nova services
3) try to create a new aggregate: `nova aggregate-create my-aggregate2`
4) look at the message bus (or the logs if you set the notification driver to log first)

nova version from master: bc4c91e7be8e4c52506a8f970078ecee6655ee82
config: default devstack +

notify_on_api_faults = True

driver = log

tags: added: notification
Balazs Gibizer (balazs-gibizer) wrote :

A mail posted to the ML with more implementation details http://lists.openstack.org/pipermail/openstack-dev/2017-June/118639.html

Sean Dague (sdague) on 2017-06-28
Changed in nova:
status: New → Confirmed
importance: Undecided → Medium
tags: added: openstack-version.newton

Fix proposed to branch: master
Review: https://review.openstack.org/505164

Changed in nova:
assignee: nobody → Balazs Gibizer (balazs-gibizer)
status: Confirmed → In Progress
Matt Riedemann (mriedem) wrote :

To be clear, this is just legacy notifications, not versioned notifications.

tags: added: notifications
removed: notification
Changed in nova:
assignee: Balazs Gibizer (balazs-gibizer) → Matt Riedemann (mriedem)
Changed in nova:
assignee: Matt Riedemann (mriedem) → Stephen Finucane (stephenfinucane)
Changed in nova:
assignee: Stephen Finucane (stephenfinucane) → Balazs Gibizer (balazs-gibizer)

Reviewed: https://review.openstack.org/505164
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=0d4c3cc65b78bafef69afc45cb156afe38f857c3
Submitter: Jenkins
Branch: master

commit 0d4c3cc65b78bafef69afc45cb156afe38f857c3
Author: Balazs Gibizer <email address hidden>
Date: Mon Sep 18 16:53:32 2017 +0200

    Remove dead code of api.fault notification sending

    Based on the description of the notify_on_api_faults config option [1]
    and the code that uses it [2] nova sends and api.fault notification
    if the nova-api service encounters an unhandle exception.
    There is a FaultWrapper class [3] added to the pipeline of the REST
    request which catches every exception and calls the notification sending.

    Based on some debugging in devstack this FaultWrapper never catches any
    exception. Every REST API method is decorated with expected_errors
    decorator [5] which as a last resort translate the unexpected exception
    to HTTPInternalServerError. In the wsgi stack the actual REST api call is
    guarded with ResourceExceptionHandler context manager [7] which translates
    HTTPException to a Fault [8]. Then Fault is catched and translated to
    the REST response [7]. This way the exception never propagates back to
    the FaultWrapper and therefore the api.fault notification is never emitted.

    Based on the git history of the expected_errors decorator this notification
    was never emitted for v2.1 API and as the v2.0 API now supported with the
    same codebase than v2.1 it is not emitted for v2.0 calls either. As nobody
    reported a bug I assume that nobody tried to use this notification for a very
    long time. Therefore instead of fixing this bug this patch propses to remove
    the dead code.

    See a bit more detailed description on the ML [9].

    [1] https://github.com/openstack/nova/blob/0aeaa2bce8ad15bc2f28e00b30d688514b992e13/nova/conf/notifications.py#L49
    [2] https://github.com/openstack/nova/blob/0aeaa2bce8ad15bc2f28e00b30d688514b992e13/nova/notifications/base.py#L84
    [3] https://github.com/openstack/nova/blob/0aeaa2bce8ad15bc2f28e00b30d688514b992e13/nova/api/openstack/__init__.py#L78
    [5] https://github.com/openstack/nova/blob/0aeaa2bce8ad15bc2f28e00b30d688514b992e13/nova/api/openstack/extensions.py#L325
    [7] https://github.com/openstack/nova/blob/0aeaa2bce8ad15bc2f28e00b30d688514b992e13/nova/api/openstack/wsgi.py#L637
    [8] https://github.com/openstack/nova/blob/0aeaa2bce8ad15bc2f28e00b30d688514b992e13/nova/api/openstack/wsgi.py#L418
    [9] http://lists.openstack.org/pipermail/openstack-dev/2017-June/118639.html

    Change-Id: I608b6ebdc69d31eb2a11ac6479fa4f2e8c20f7d1
    Closes-Bug: #1699115

Changed in nova:
status: In Progress → Fix Released

This issue was fixed in the openstack/nova development milestone.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers