initialize_connection failure can result in 'exceptions.TypeError' object has no attribute 'code'

[Carl Pecinovsky]

Doing an initialize_connection() results in the following partial code stack:

2016-03-29 06:51:06.882 25713 ERROR nova.compute.manager [instance: 4c8d25b3-b136-4c72-a381-4b80dcca5c49] File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 232, in wrapper
2016-03-29 06:51:06.882 25713 ERROR nova.compute.manager [instance: 4c8d25b3-b136-4c72-a381-4b80dcca5c49] res = method(self, ctx, *args, **kwargs)
2016-03-29 06:51:06.882 25713 ERROR nova.compute.manager [instance: 4c8d25b3-b136-4c72-a381-4b80dcca5c49] File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 259, in wrapper
2016-03-29 06:51:06.882 25713 ERROR nova.compute.manager [instance: 4c8d25b3-b136-4c72-a381-4b80dcca5c49] res = method(self, ctx, volume_id, *args, **kwargs)
2016-03-29 06:51:06.882 25713 ERROR nova.compute.manager [instance: 4c8d25b3-b136-4c72-a381-4b80dcca5c49] File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 437, in initialize_connection
2016-03-29 06:51:06.882 25713 ERROR nova.compute.manager [instance: 4c8d25b3-b136-4c72-a381-4b80dcca5c49] 'code': exc.code})
2016-03-29 06:51:06.882 25713 ERROR nova.compute.manager [instance: 4c8d25b3-b136-4c72-a381-4b80dcca5c49] AttributeError: 'exceptions.TypeError' object has no attribute 'code'

Here is the method:

    @translate_volume_exception
    def initialize_connection(self, context, volume_id, connector):
        try:
            connection_info = cinderclient(
                context).volumes.initialize_connection(volume_id, connector)
            connection_info['connector'] = connector
            return connection_info
        except cinder_exception.ClientException as ex:
            with excutils.save_and_reraise_exception():
                LOG.error(_LE('Initialize connection failed for volume '
                              '%(vol)s on host %(host)s. Error: %(msg)s '
                              'Code: %(code)s. Attempting to terminate '
                              'connection.'),
                          {'vol': volume_id,
                           'host': connector.get('host'),
                           'msg': six.text_type(ex),
                           'code': ex.code})
                try:
                    self.terminate_connection(context, volume_id, connector)
                except Exception as exc:
                    LOG.error(_LE('Connection between volume %(vol)s and host '
                                  '%(host)s might have succeeded, but attempt '
                                  'to terminate connection has failed. '
                                  'Validate the connection and determine if '
                                  'manual cleanup is needed. Error: %(msg)s '
                                  'Code: %(code)s.'),
                              {'vol': volume_id,
                               'host': connector.get('host'),
                               'msg': six.text_type(exc),
                               'code': exc.code}) <---------- blows up

    @translate_volume_exception
    def terminate_connection(self, context, volume_id, connector):
        return cinderclient(context).volumes.terminate_connection(volume_id,
                                                                  connector)

Presumably the issue has to do with the @translate_volume_exception decorator on terminate_connection()....so when that fails, the exception that is re-raised is not constructed with a 'code' value (?)

[Matt Riedemann (mriedem)]

What version of nova is this? Mitaka? It probably doesn't really matter, the issue is here as you pointed out:

except Exception as exc:
                    LOG.error(_LE('Connection between volume %(vol)s and host '
                                  '%(host)s might have succeeded, but attempt '
                                  'to terminate connection has failed. '
                                  'Validate the connection and determine if '
                                  'manual cleanup is needed. Error: %(msg)s '
                                  'Code: %(code)s.'),
                              {'vol': volume_id,
                               'host': connector.get('host'),
                               'msg': six.text_type(exc),
                               'code': exc.code}) <---------- blows up

exc in this case is a TypeError, which doesn't have a code on it. That code is probably expecting a CinderClientException or NovaException, but something else breaks and we get a TypeError instead.

[Carl Pecinovsky (csky)]

Yes, Mitaka.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/300751

[Wenzhi Yu (yuywz)]

@Diana, per the bug history, it seems you already assigned this bug to you before I did... sorry for that but when I took the bug, it shows 'nobody' in the 'assigned to' filed, it must be some network delay...

[Diana Clarke (diana-clarke)]

@Wenzhi No worries :) I figured that's what happened. Cheers, --diana

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/300751
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2135a3f55ff2222ec9c0fe7064aa339a430af4ee
Submitter: Jenkins
Branch: master

commit 2135a3f55ff2222ec9c0fe7064aa339a430af4ee
Author: Wenzhi Yu <[e-mail]>
Date: Sat Apr 2 23:34:22 2016 +0800

    Check if a exception has a code on it before read the code

    In 'nova.volume.cinder.API.initialize_connection' method, all exceptions
    threw by 'terminate_connection' will be caught and logged. When log the
    exceptions, the code try to record the 'code' attribute of the exception
    since the code is expecting a CinderClientException or NovaException[1].
    But if the some else exception(like TypeError) which doesn't have a code
    on it was threw up, we will got a AttributeError.

    This commit add logic to check if the exception has a code on it before
    try to read it.

    [1]https://github.com/openstack/nova/blob/13.0.0.0rc3/nova/volume/cinder.py#L437

    Change-Id: I42fd2f2b77c41a60dfaf0cc882a344596d50daf5
    Closes-Bug: #1564551

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/303757

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/303880

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/303757
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=9ecf79970d000ad48772bd3f618d9454b57b5985
Submitter: Jenkins
Branch: stable/mitaka

commit 9ecf79970d000ad48772bd3f618d9454b57b5985
Author: Wenzhi Yu <[e-mail]>
Date: Sat Apr 2 23:34:22 2016 +0800

    Check if a exception has a code on it before read the code

    In 'nova.volume.cinder.API.initialize_connection' method, all exceptions
    threw by 'terminate_connection' will be caught and logged. When log the
    exceptions, the code try to record the 'code' attribute of the exception
    since the code is expecting a CinderClientException or NovaException[1].
    But if the some else exception(like TypeError) which doesn't have a code
    on it was threw up, we will got a AttributeError.

    This commit add logic to check if the exception has a code on it before
    try to read it.

    [1]https://github.com/openstack/nova/blob/13.0.0.0rc3/nova/volume/cinder.py#L437

    Change-Id: I42fd2f2b77c41a60dfaf0cc882a344596d50daf5
    Closes-Bug: #1564551
    (cherry picked from commit 2135a3f55ff2222ec9c0fe7064aa339a430af4ee)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/303880
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5e12cd8756e3b2871bd9ba5321528cb4fc5854f9
Submitter: Jenkins
Branch: stable/liberty

commit 5e12cd8756e3b2871bd9ba5321528cb4fc5854f9
Author: Wenzhi Yu <[e-mail]>
Date: Sat Apr 2 23:34:22 2016 +0800

    Check if a exception has a code on it before read the code

    In 'nova.volume.cinder.API.initialize_connection' method, all exceptions
    threw by 'terminate_connection' will be caught and logged. When log the
    exceptions, the code try to record the 'code' attribute of the exception
    since the code is expecting a CinderClientException or NovaException[1].
    But if the some else exception(like TypeError) which doesn't have a code
    on it was threw up, we will got a AttributeError.

    This commit add logic to check if the exception has a code on it before
    try to read it.

    [1]https://github.com/openstack/nova/blob/13.0.0.0rc3/nova/volume/cinder.py#L437

    Change-Id: I42fd2f2b77c41a60dfaf0cc882a344596d50daf5
    Closes-Bug: #1564551
    (cherry picked from commit 2135a3f55ff2222ec9c0fe7064aa339a430af4ee)

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/nova 14.0.0.0b1 development milestone.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/nova 12.0.4 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/nova 13.1.0 release.