Nova and quantum don't close qpid connections after certain error conditions

Bug #1172922 reported by Brad Pokorny on 2013-04-25
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Cinder
High
Unassigned
Grizzly
High
Unassigned
OpenStack Compute (nova)
High
Michael Still
Grizzly
High
Ben Nemec
neutron
High
Gary Kotton
Grizzly
High
Gary Kotton
oslo-incubator
High
Gary Kotton

Bug Description

In conditions where nova gets a connection to qpid but there is a problem at the application layer (such as a bad qpid password is specified), nova will continuously retry the connection without closing previous connections.

Steps to reproduce:
- Preconditions: At least 1 nova compute node using qpid for messaging, with authentication turned on.
- Specify a wrong qpid password in nova.conf on the compute node
- nova will continuously retry with the wrong password and print errors such as this to the compute log:
2013-04-25 16:37:52.269 ERROR nova.openstack.common.rpc.impl_qpid [req-1d15f33c-5b2d-4ee1-aaa1-ab0140a56608 None None] Unable to connect to AMQP server: connection-forced: Authentication failed(320). Sleeping 60 seconds
- Each time nova retries the connection, it will create another connection to qpid and not close the previous connections.

I think this exception case needs to be updated in https://github.com/openstack/nova/blob/master/nova/openstack/common/rpc/impl_qpid.py to close the connection:

            except qpid_exceptions.ConnectionError, e:
                msg_dict = dict(e=e, delay=delay)
                msg = _("Unable to connect to AMQP server: %(e)s. "
                        "Sleeping %(delay)s seconds") % msg_dict
                LOG.error(msg)
                time.sleep(delay)
                delay = min(2 * delay, 60)

Note that quantum uses the same code for qpid.

Mark McClain (markmcclain) wrote :

This must looked at in Oslo first before changes can be included in Quantum.

Gary Kotton (garyk) on 2013-04-27
tags: added: oslo

Fix proposed to branch: master
Review: https://review.openstack.org/27894

Changed in oslo:
assignee: nobody → Gary Kotton (garyk)
status: New → In Progress

Reviewed: https://review.openstack.org/27894
Committed: http://github.com/openstack/oslo-incubator/commit/0c9047cc334578f9f4974c3bc006ba9bc62814d2
Submitter: Jenkins
Branch: master

commit 0c9047cc334578f9f4974c3bc006ba9bc62814d2
Author: Gary Kotton <email address hidden>
Date: Wed May 1 07:52:06 2013 +0000

    Ensure that qpid connection is closed.

    Fixes bug 1172922

    When reestablishing a session and a connection already exists the
    existing connection will be closed prior to opening the new one.

    Change-Id: I9a25800fe4b56eba89d550e3461ce095bf902f0a

Changed in oslo:
status: In Progress → Fix Committed

Fix proposed to branch: master
Review: https://review.openstack.org/27909

Changed in quantum:
assignee: nobody → Gary Kotton (garyk)
status: New → In Progress
Gary Kotton (garyk) on 2013-05-01
tags: added: grizzly-backport-potential
Mark McLoughlin (markmc) on 2013-05-01
Changed in oslo:
importance: Undecided → High
Changed in nova:
importance: Undecided → High
Changed in quantum:
importance: Undecided → High

Reviewed: https://review.openstack.org/27909
Committed: http://github.com/openstack/quantum/commit/72e2a2515b241d8abfaa7d8394b611224027f7dd
Submitter: Jenkins
Branch: master

commit 72e2a2515b241d8abfaa7d8394b611224027f7dd
Author: Gary Kotton <email address hidden>
Date: Wed May 1 13:47:59 2013 +0000

    Update RPC code from OSLO

    Fixes bug 1172922

    Change-Id: Ieb5f58fe3d2c879bc71f4241288e48e35ab54366

Changed in quantum:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/27911
Committed: http://github.com/openstack/oslo-incubator/commit/11b7d88c8a6ee7df1eb14d75b35b1c516d1bd8f2
Submitter: Jenkins
Branch: stable/grizzly

commit 11b7d88c8a6ee7df1eb14d75b35b1c516d1bd8f2
Author: Gary Kotton <email address hidden>
Date: Wed May 1 07:52:06 2013 +0000

    Ensure that qpid connection is closed.

    Fixes bug 1172922

    When reestablishing a session and a connection already exists the
    existing connection will be closed prior to opening the new one.

    Change-Id: I9a25800fe4b56eba89d550e3461ce095bf902f0a

tags: added: in-stable-grizzly
Changed in quantum:
milestone: none → havana-1
Gary Kotton (garyk) on 2013-05-06
tags: removed: grizzly-backport-potential
Gary Kotton (garyk) on 2013-05-06
tags: added: grizzly-backport-potential
Michael Still (mikal) wrote :

Does this just need an oslo sync for nova now? Or is there more to it than that?

Changed in nova:
status: New → Triaged
Gary Kotton (garyk) wrote :

Hi,
This just requires a sync of the OSLO RPC qpid code. This was done in Quantum in https://review.openstack.org/#/c/28350/
Thanks
Gary

Reviewed: https://review.openstack.org/28350
Committed: http://github.com/openstack/quantum/commit/5a86b4e45ff32bd8a8d728122a769cc6c2cfe51e
Submitter: Jenkins
Branch: stable/grizzly

commit 5a86b4e45ff32bd8a8d728122a769cc6c2cfe51e
Author: Gary Kotton <email address hidden>
Date: Mon May 6 21:39:06 2013 +0000

    OSLO updates for stable

    Fixes bug 1172922
    Fixes bug 1159889
    Fixes bug 1158319

    Change-Id: I6257b5636b7b98d26ee43cbbf6087d9279a5b97c

Gary Kotton (garyk) on 2013-05-13
tags: removed: grizzly-backport-potential
Thierry Carrez (ttx) on 2013-05-29
Changed in quantum:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2013-05-30
Changed in oslo:
milestone: none → havana-1
status: Fix Committed → Fix Released
Alan Pevec (apevec) on 2013-07-09
tags: removed: in-stable-grizzly
Alan Pevec (apevec) wrote :
Changed in nova:
status: Triaged → Fix Committed
assignee: nobody → Michael Still (mikalstill)
status: Fix Committed → Fix Released
Alan Pevec (apevec) on 2013-07-09
Changed in nova:
milestone: none → havana-1

Reviewed: https://review.openstack.org/33476
Committed: http://github.com/openstack/nova/commit/d2cb4b25f7226cd2051d38905942e515d257931c
Submitter: Jenkins
Branch: stable/grizzly

commit d2cb4b25f7226cd2051d38905942e515d257931c
Author: Ben Nemec <email address hidden>
Date: Tue Jun 18 16:59:04 2013 +0000

    Sync RPC module from Oslo

    Includes backport of change for bug 1175808
    Fixes bug 1172922 and bug 1122763

    Change-Id: Ib52e9458a9db01e2c8fd2a6d84010733f85980f8

Eric Harney (eharney) on 2013-08-26
Changed in cinder:
milestone: none → havana-3
Eric Harney (eharney) on 2013-08-26
Changed in cinder:
importance: Undecided → High

Fix proposed to branch: master
Review: https://review.openstack.org/43711

Changed in cinder:
status: New → In Progress
assignee: nobody → Xavier Queralt (xqueralt)

Reviewed: https://review.openstack.org/43711
Committed: http://github.com/openstack/cinder/commit/4576ea44aa51703d7ed3b036a044dbd911d6f3e9
Submitter: Jenkins
Branch: master

commit 4576ea44aa51703d7ed3b036a044dbd911d6f3e9
Author: Xavier Queralt <email address hidden>
Date: Mon Aug 26 16:11:34 2013 +0200

    Ensure that qpid connection is closed (from oslo)

    This is commit a9bc62814d in oslo-incubator

    Fixes bug 1172922

    When reestablishing a session and a connection already exists the
    existing connection will be closed prior to opening the new one.

    Change-Id: I9a25800fe4b56eba89d550e3461ce095bf902f0a

Changed in cinder:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2013-09-05
Changed in cinder:
status: Fix Committed → Fix Released

I will be out of the office starting 2013-09-05 and will not return until
2013-09-15.

I will be on my marriage leave from 9/5 to 9/15, for any urgent issue
please call me before 9/7.

For daily work, please ask my scrum master Zhu Zhu for help.
For glance issue, please ask glance SME Feilong Wang for help.
For defect report, there will be no report next week.

Reviewed: https://review.openstack.org/43713
Committed: http://github.com/openstack/cinder/commit/3917e9dfa5549678296f9b4295369e584802a04d
Submitter: Jenkins
Branch: stable/grizzly

commit 3917e9dfa5549678296f9b4295369e584802a04d
Author: Xavier Queralt <email address hidden>
Date: Mon Aug 26 16:11:34 2013 +0200

    Ensure that qpid connection is closed (from oslo)

    This is commit a9bc62814d in oslo-incubator

    Fixes bug 1172922

    When reestablishing a session and a connection already exists the
    existing connection will be closed prior to opening the new one.

    Change-Id: I9a25800fe4b56eba89d550e3461ce095bf902f0a

tags: added: in-stable-grizzly
Thierry Carrez (ttx) on 2013-10-17
Changed in oslo:
milestone: havana-1 → 2013.2
Thierry Carrez (ttx) on 2013-10-17
Changed in cinder:
milestone: havana-3 → 2013.2
Thierry Carrez (ttx) on 2013-10-17
Changed in neutron:
milestone: havana-1 → 2013.2
Changed in nova:
milestone: havana-1 → 2013.2
Alan Pevec (apevec) on 2014-03-30
tags: removed: in-stable-grizzly
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers