notifications are not being recevied since mitaka

[Adam Gandelman]

our liberty release was 1 week behind neutron. in that week, we had to pin our gate jobs to stable/liberty for other projects because at some point since mitaka opened, something changed in neutron that prevents events from hitting astara.

have not yet been able to track down the change but am in the process of bisecting now.

will update this bug with details once i find more

[Adam Gandelman (gandelman-a)]

It looks like this is actually due to a change in oslo.messaging, specifically on the neutron side. With oslo.messaging 2.5.0 (liberty) neutron notifications are sent by neutron and received by astara, with 2.6.0 they are not. trunk/mitaka is running /w 2.7.0 ATM

[Adam Gandelman (gandelman-a)]

Bisected this back to https://review.openstack.org/#/c/227374/ Prior to that, neutron was sending out event notifications. Worth noting that the astara devstack plugin sets notification_driver = neutron.openstack.common.notifier.rpc_notifier in neutron.conf. devstack sets this to 'messaging' in all other projects. Attempting to do that in neutron doesn't seem to help and digging closer it looks like stevedore is unable to load any of the notification drivers. Not sure if this is a bug in 227374 and no projects are able to load notification drivers from oslo_messaging, if neutron is a special case and needs something else, or what...

[Davanum Srinivas (DIMS) (dims-v)]

Neutron has:
    neutron.openstack.common.notifier.rpc_notifier2 = oslo_messaging.notify._impl_messaging:MessagingV2Driver
    neutron.openstack.common.notifier.rpc_notifier = oslo_messaging.notify._impl_messaging:MessagingDriver

oslo.messaging has:
    messagingv2 = oslo_messaging.notify.messaging:MessagingV2Driver
    messaging = oslo_messaging.notify.messaging:MessagingDriver

note that _impl means internal implementation :) but yeah. that would cause you to fail.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/242775

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/242775
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=225663888e71d44e9732951f515da925b8799b74
Submitter: Jenkins
Branch: master

commit 225663888e71d44e9732951f515da925b8799b74
Author: Davanum Srinivas <email address hidden>
Date: Sat Nov 7 14:14:05 2015 -0500

    Fix notification driver package

    In oslo.messaging change:
    Ifb96c2ae9868426cac2700bf4917c27c02c90b15

    the private package for the messaging driver was changed
    to a public package. So we should fix our reference here
    in our setup.cfg

    Closes-Bug: #1513630

    Change-Id: I6b066f4f1b6c6485cb67ca5f8a2a17359521c264

[Adam Gandelman (gandelman-a)]

stable/liberty is now getting bit by this, too. Its still specifying the private paths in setup.cfg, but stable/liberty's oslo.messaging requirement is uncapped, pulling in the newer versions /w modified paths.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/247276

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: stable/liberty
Review: https://review.openstack.org/247276
Reason: We're going to create an alias to the old name in oslo.messaging and release that, then blacklist oslo.messaging 2.6.0->the release that has the alias in stable/liberty g-r.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/247735

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/247906

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/247735
Committed: https://git.openstack.org/cgit/openstack/oslo.messaging/commit/?id=15cd99050c6d2714b90059e0faad9f9e3409eaaa
Submitter: Jenkins
Branch: master

commit 15cd99050c6d2714b90059e0faad9f9e3409eaaa
Author: Matt Riedemann <email address hidden>
Date: Thu Nov 19 10:42:34 2015 -0800

    Provide alias to oslo_messaging.notify._impl_messaging

    Ifb96c2ae9868426cac2700bf4917c27c02c90b15 moved the _impl_messaging
    module to oslo_messaging.notify.messaging which breaks neutron.

    Neutron is fixed on master for mitaka but neutron on stable/liberty
    is broken, and changing neutron on stable/liberty to use the new
    path would require a global-requirements minimum version bump for
    oslo.messaging to 2.6.0, which we want to avoid for people that have
    already shipped liberty.

    So provide an alias to the moved module so neutron in stable/liberty
    continues to work. We deprecate the module so consumers know they
    need to upgrade and move off this. We may need to cap oslo.messaging
    in global-requirements on stable/liberty at some point when we remove
    the deprecated alias module.

    Change-Id: I29453e0fbf30b0a571c2b1afc7cc81d1a11535f0
    Closes-Bug: #1513630

[Matt Riedemann (mriedem)]

Here is the fix to neutron on stable/liberty: https://review.openstack.org/#/c/249367/

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/neutron 8.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/247906
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=fe987b5dd5e2b53b71c2bf7afdc46359b4c4d574
Submitter: Jenkins
Branch: master

commit fe987b5dd5e2b53b71c2bf7afdc46359b4c4d574
Author: Matt Riedemann <email address hidden>
Date: Thu Nov 19 18:43:07 2015 -0800

    Set timetable for removal of oslo.messaging.notify.drivers

    Icehouse is dead and gone, at least upstream. These special driver
    registrations are not tested in the gate-tempest-dsvm-neutron-full job
    which means they are also not tested in requirements constraints jobs.

    oslo.messaging 2.6.0 broke these already by removing the internal modules,
    which was fixed in o.m 3.0.0 with (deprecated) alias modules.

    The minimum required version of o.m in mitaka is currently greater than
    2.6.1, so we're OK to remove these once stable/mitaka is our oldest
    supported branch. So add a TODO to remove these once liberty-eol happens.

    Proper configuration for notification drivers happens through the
    config file using the oslo_messaging options:

    http://docs.openstack.org/developer/oslo.messaging/opts.html

    Adds a release note for the deprecation and timetable.

    Change-Id: I2c04117b0f531fe3604e74401d0ffd8d8d3dd308
    Related-Bug: #1513630

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: feature/pika
Review: https://review.openstack.org/257373

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/257373
Committed: https://git.openstack.org/cgit/openstack/oslo.messaging/commit/?id=cc0f8cc8a9ff25c9fb081cac5366c12a0c06ec53
Submitter: Jenkins
Branch: feature/pika

commit a5d78891745b6b9e5827271dc305f00acae1392f
Author: OpenStack Proposal Bot <email address hidden>
Date: Fri Dec 11 15:24:05 2015 +0000

    Updated from global requirements

    Change-Id: Ifd78016c067740477a82dbe06d74d5944ba91893

commit 17ccb2306d03a74304c57d31716a54ba2b3b4311
Author: Mehdi Abaakouk <email address hidden>
Date: Fri Dec 11 10:59:54 2015 +0100

    Move to debug a too verbose log

    When a client is gone (died/restart) and somes replies cannot be sent because
    the the exchange of this client will never comeback. We log one message per
    reply every 0.25 messages during 60 seconds. When the only useful log
    is the one where we decide to drop this replies.

    This change moves the less important message to debug level.

    Change-Id: I508787c0db4dcec2c0027b89eb4e65c4f98022b9
    Related-bug: #1524418

commit 46daf858144202a072c4bf8580aeafec11d20e13
Author: Davanum Srinivas <email address hidden>
Date: Fri Dec 11 11:04:13 2015 +0300

    Cleanup parameter docstrings

    Change-Id: I301fdd51446bf0c0a6dd0d05b26da0556db8367d

commit 3ee86964fa460882d8fcac8686edd0e6bfb12008
Author: Mehdi Abaakouk <email address hidden>
Date: Wed Dec 9 19:37:40 2015 +0100

    Revert "default of kombu_missing_consumer_retry_timeout"

    This reverts commit 8c03a6db6c0396099e7425834998da5478a1df7c.

    Closes-bug: #1524418
    Change-Id: I35538a6c15d6402272e4513bc1beaa537b0dd7b9

commit e72599435c59c09277a9da7686b32aa4f9df7ba4
Author: Mehdi Abaakouk <email address hidden>
Date: Wed Dec 9 18:49:19 2015 +0100

    Don't trigger error_callback for known exc

    When AMQPDestinationNotFound is raised, we must not
    call the error_callback method. The exception is logged
    only if needed in upper layer (amqpdriver.py).

    Related-bug: #1524418

    Change-Id: Ic1ddec2d13172532dbaa572d04a4c22c97ac4fe7

commit 185693a6ed57e02b2f94b0fb8f14a91471605969
Author: Mehdi Abaakouk <email address hidden>
Date: Wed Dec 9 11:23:52 2015 +0100

    Improves comment

    Change-Id: Idc8002e6d622435aac48304857985c0f82be3e32

commit 148e8380ce1cc4f60716300b95104aaa2cf8c543
Author: Mehdi Abaakouk <email address hidden>
Date: Fri Dec 4 14:57:03 2015 +0100

    Fix reconnection when heartbeat is missed

    When a heartbeat is missing we call ensure_connection()
    that runs a dummy method to trigger the reconnection
    code in kombu. But also the code is triggered only if the
    channel is None.

    In case of the heartbeat threads we didn't reset the channel
    before reconnecting, so the dummy method doesn't do anything.

    This change sets the channel to None to ensure the connection
    is reestablished before the dummy method is run.

    Also it replaces the dummy method by checking the kombu connection
    object. So we are sure the connection is reestablished.

    Change-Id: I39f8cd23c5a5498e6f4c1aa3236ed27f3b5d7c9a
    Closes-bug: #1493890

commit 05002...