Nova and quantum don't close qpid connections after certain error conditions

[Brad Pokorny]

In conditions where nova gets a connection to qpid but there is a problem at the application layer (such as a bad qpid password is specified), nova will continuously retry the connection without closing previous connections.

Steps to reproduce:
- Preconditions: At least 1 nova compute node using qpid for messaging, with authentication turned on.
- Specify a wrong qpid password in nova.conf on the compute node
- nova will continuously retry with the wrong password and print errors such as this to the compute log:
2013-04-25 16:37:52.269 ERROR nova.openstack.common.rpc.impl_qpid [req-1d15f33c-5b2d-4ee1-aaa1-ab0140a56608 None None] Unable to connect to AMQP server: connection-forced: Authentication failed(320). Sleeping 60 seconds
- Each time nova retries the connection, it will create another connection to qpid and not close the previous connections.

I think this exception case needs to be updated in https://github.com/openstack/nova/blob/master/nova/openstack/common/rpc/impl_qpid.py to close the connection:

            except qpid_exceptions.ConnectionError, e:
                msg_dict = dict(e=e, delay=delay)
                msg = _("Unable to connect to AMQP server: %(e)s. "
                        "Sleeping %(delay)s seconds") % msg_dict
                LOG.error(msg)
                time.sleep(delay)
                delay = min(2 * delay, 60)

Note that quantum uses the same code for qpid.

[Mark McClain (markmcclain)]

This must looked at in Oslo first before changes can be included in Quantum.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/27894

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/27894
Committed: http://github.com/openstack/oslo-incubator/commit/0c9047cc334578f9f4974c3bc006ba9bc62814d2
Submitter: Jenkins
Branch: master

commit 0c9047cc334578f9f4974c3bc006ba9bc62814d2
Author: Gary Kotton <email address hidden>
Date: Wed May 1 07:52:06 2013 +0000

    Ensure that qpid connection is closed.

    Fixes bug 1172922

    When reestablishing a session and a connection already exists the
    existing connection will be closed prior to opening the new one.

    Change-Id: I9a25800fe4b56eba89d550e3461ce095bf902f0a

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/27909

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/27911

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/27909
Committed: http://github.com/openstack/quantum/commit/72e2a2515b241d8abfaa7d8394b611224027f7dd
Submitter: Jenkins
Branch: master

commit 72e2a2515b241d8abfaa7d8394b611224027f7dd
Author: Gary Kotton <email address hidden>
Date: Wed May 1 13:47:59 2013 +0000

    Update RPC code from OSLO

    Fixes bug 1172922

    Change-Id: Ieb5f58fe3d2c879bc71f4241288e48e35ab54366

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/27911
Committed: http://github.com/openstack/oslo-incubator/commit/11b7d88c8a6ee7df1eb14d75b35b1c516d1bd8f2
Submitter: Jenkins
Branch: stable/grizzly

commit 11b7d88c8a6ee7df1eb14d75b35b1c516d1bd8f2
Author: Gary Kotton <email address hidden>
Date: Wed May 1 07:52:06 2013 +0000

    Ensure that qpid connection is closed.

    Fixes bug 1172922

    When reestablishing a session and a connection already exists the
    existing connection will be closed prior to opening the new one.

    Change-Id: I9a25800fe4b56eba89d550e3461ce095bf902f0a

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/28350

[Michael Still (mikal)]

Does this just need an oslo sync for nova now? Or is there more to it than that?

[Gary Kotton (garyk)]

Hi,
This just requires a sync of the OSLO RPC qpid code. This was done in Quantum in https://review.openstack.org/#/c/28350/
Thanks
Gary

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/28350
Committed: http://github.com/openstack/quantum/commit/5a86b4e45ff32bd8a8d728122a769cc6c2cfe51e
Submitter: Jenkins
Branch: stable/grizzly

commit 5a86b4e45ff32bd8a8d728122a769cc6c2cfe51e
Author: Gary Kotton <email address hidden>
Date: Mon May 6 21:39:06 2013 +0000

    OSLO updates for stable

    Fixes bug 1172922
    Fixes bug 1159889
    Fixes bug 1158319

    Change-Id: I6257b5636b7b98d26ee43cbbf6087d9279a5b97c

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/33476

[Alan Pevec (apevec)]

Fix merged to nova (master):
https://github.com/openstack/nova/commit/b4826d85c25a56ad95ffb76c467cdb459daba0c4

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/33476
Committed: http://github.com/openstack/nova/commit/d2cb4b25f7226cd2051d38905942e515d257931c
Submitter: Jenkins
Branch: stable/grizzly

commit d2cb4b25f7226cd2051d38905942e515d257931c
Author: Ben Nemec <email address hidden>
Date: Tue Jun 18 16:59:04 2013 +0000

    Sync RPC module from Oslo

    Includes backport of change for bug 1175808
    Fixes bug 1172922 and bug 1122763

    Change-Id: Ib52e9458a9db01e2c8fd2a6d84010733f85980f8

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/43713

[Xavier Queralt (xqueralt-deactivatedaccount)]

Fix proposed to branch: master
Review: https://review.openstack.org/43711

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/43711
Committed: http://github.com/openstack/cinder/commit/4576ea44aa51703d7ed3b036a044dbd911d6f3e9
Submitter: Jenkins
Branch: master

commit 4576ea44aa51703d7ed3b036a044dbd911d6f3e9
Author: Xavier Queralt <email address hidden>
Date: Mon Aug 26 16:11:34 2013 +0200

    Ensure that qpid connection is closed (from oslo)

    This is commit a9bc62814d in oslo-incubator

    Fixes bug 1172922

    When reestablishing a session and a connection already exists the
    existing connection will be closed prior to opening the new one.

    Change-Id: I9a25800fe4b56eba89d550e3461ce095bf902f0a

[Yang Yu (yuyangbj)]

I will be out of the office starting 2013-09-05 and will not return until
2013-09-15.

I will be on my marriage leave from 9/5 to 9/15, for any urgent issue
please call me before 9/7.

For daily work, please ask my scrum master Zhu Zhu for help.
For glance issue, please ask glance SME Feilong Wang for help.
For defect report, there will be no report next week.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/43713
Committed: http://github.com/openstack/cinder/commit/3917e9dfa5549678296f9b4295369e584802a04d
Submitter: Jenkins
Branch: stable/grizzly

commit 3917e9dfa5549678296f9b4295369e584802a04d
Author: Xavier Queralt <email address hidden>
Date: Mon Aug 26 16:11:34 2013 +0200

    Ensure that qpid connection is closed (from oslo)

    This is commit a9bc62814d in oslo-incubator

    Fixes bug 1172922

    When reestablishing a session and a connection already exists the
    existing connection will be closed prior to opening the new one.

    Change-Id: I9a25800fe4b56eba89d550e3461ce095bf902f0a