oslo.messaging hangs if SASL authentication fails

[Matthew Booth]

I appear to have an invalid sasl configuration locally. This is a separate issue, however, it causes a couple of tests to fail *by timing out*. These tests are:

  oslo_messaging.tests.test_amqp_driver.TestCyrusAuthentication.test_authentication_default_username
  oslo_messaging.tests.test_amqp_driver.TestCyrusAuthentication.test_authentication_ok

Checking the failure tests, notice that the failure case is:

        self.assertRaises(oslo_messaging.MessagingTimeout,
                          driver.send,
                          target, {"context": True},
                          {"method": "echo"},
                          wait_for_reply=True,
                          timeout=2.0)

Sure enough, if you remove the timeout argument it also hangs. I would expect this to fail rather than hang.

[Ken Giusti (kgiusti)]

Just as an aside - I believe this behavior is not limited to the amqp1 driver.

Playing a bit with oslo.messaging 2.7.0 and rabbitmq, I get the same hang if I configure an incorrect password. The client just keeps attempting to reconnect indefinitely.

Unlike the amqp1 driver, the rabbitmq driver supports an optional "retry" parameter. This integer parameter limits the total number of connection attempts, and will raise a MessageDeliveryFailure exception if the connection does not succeed before the retry limit is hit.

While this can be used to prevent infinite retries, I don't think it's the proper way oslo.messaging should deal with authentication errors. If a client can't authenticate with any of its configured hosts there's no resolving that without intervention from meatspace. Raising a hard failure would get the operator's attention way faster than client hangs and log messages.

I've intended to add 'retry' to the amqp1 driver (https://bugs.launchpad.net/oslo.messaging/+bug/1434540) fyi.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/285880
Committed: https://git.openstack.org/cgit/openstack/oslo.messaging/commit/?id=97655c1fe633f6b1340ce899bb1ad65284926399
Submitter: Jenkins
Branch: master

commit 97655c1fe633f6b1340ce899bb1ad65284926399
Author: Davanum Srinivas <email address hidden>
Date: Sun Feb 28 23:14:12 2016 -0500

    Fail quickly if there on bad password

    Inspired by:
    https://bugs.launchpad.net/oslo.messaging/+bug/1508512/comments/1

    Add authentication_failure_close capability on the client to
    tell the server that it can send us connection.close command
    indicating ACCESS_REFUSED as the reasona. If this capability is
    absent then authentication failures are reported in the legacy
    fashion: by abruptly closing the network connection.

    Partial-Bug: #1508512
    Change-Id: Icf6108678256e6f922620837c71f8827fe8e8c52

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/297622

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/297622
Committed: https://git.openstack.org/cgit/openstack/oslo.messaging/commit/?id=a14e1e16a71c8f11520f6cf250713fad4b81284c
Submitter: Jenkins
Branch: stable/mitaka

commit a14e1e16a71c8f11520f6cf250713fad4b81284c
Author: Davanum Srinivas <email address hidden>
Date: Sun Feb 28 23:14:12 2016 -0500

    Fail quickly if there on bad password

    Inspired by:
    https://bugs.launchpad.net/oslo.messaging/+bug/1508512/comments/1

    Add authentication_failure_close capability on the client to
    tell the server that it can send us connection.close command
    indicating ACCESS_REFUSED as the reasona. If this capability is
    absent then authentication failures are reported in the legacy
    fashion: by abruptly closing the network connection.

    Partial-Bug: #1508512
    Change-Id: Icf6108678256e6f922620837c71f8827fe8e8c52
    (cherry picked from commit 97655c1fe633f6b1340ce899bb1ad65284926399)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/409918

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/409918
Committed: https://git.openstack.org/cgit/openstack/oslo.messaging/commit/?id=9972158baa5852736ef5669efb07354513494ede
Submitter: Jenkins
Branch: master

commit 9972158baa5852736ef5669efb07354513494ede
Author: Kenneth Giusti <email address hidden>
Date: Mon Dec 12 15:32:27 2016 -0500

    [AMQP 1.0] Propagate authentication errors to caller

    When the connection fails due to an authentication issue raise a
    MessageDeliveryFailure on all pending send requests. Include the
    authentication error message in the exception.

    Change-Id: I06b40c6c480a4d082dce64801736b3d12f696c26
    Closes-Bug: 1508512

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/oslo.messaging 5.17.0 release.