Cinder Migrate from NFS fails due to nfs_mount_point_base not being provided

[Jeff Applewhite]

"/opt/stack/cinder/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data
2013-10-09 11:39:43.657 TRACE cinder.openstack.common.rpc.amqp **args)
2013-10-09 11:39:43.657 TRACE cinder.openstack.common.rpc.amqp File "/opt/stack/cinder/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch
2013-10-09 11:39:43.657 TRACE cinder.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-10-09 11:39:43.657 TRACE cinder.openstack.common.rpc.amqp File "/opt/stack/cinder/cinder/utils.py", line 808, in wrapper
2013-10-09 11:39:43.657 TRACE cinder.openstack.common.rpc.amqp return func(self, *args, **kwargs)
2013-10-09 11:39:43.657 TRACE cinder.openstack.common.rpc.amqp File "/opt/stack/cinder/cinder/volume/manager.py", line 779, in migrate_volume
2013-10-09 11:39:43.657 TRACE cinder.openstack.common.rpc.amqp self.db.volume_update(ctxt, volume_ref['id'], updates)
2013-10-09 11:39:43.657 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-10-09 11:39:43.657 TRACE cinder.openstack.common.rpc.amqp self.gen.next()
2013-10-09 11:39:43.657 TRACE cinder.openstack.common.rpc.amqp File "/opt/stack/cinder/cinder/volume/manager.py", line 772, in migrate_volume
2013-10-09 11:39:43.657 TRACE cinder.openstack.common.rpc.amqp self._migrate_volume_generic(ctxt, volume_ref, host)
2013-10-09 11:39:43.657 TRACE cinder.openstack.common.rpc.amqp File "/opt/stack/cinder/cinder/volume/manager.py", line 710, in _migrate_volume_generic
2013-10-09 11:39:43.657 TRACE cinder.openstack.common.rpc.amqp new_volume['migration_status'] = None
2013-10-09 11:39:43.657 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-10-09 11:39:43.657 TRACE cinder.openstack.common.rpc.amqp self.gen.next()
2013-10-09 11:39:43.657 TRACE cinder.openstack.common.rpc.amqp File "/opt/stack/cinder/cinder/volume/manager.py", line 690, in _migrate_volume_generic
2013-10-09 11:39:43.657 TRACE cinder.openstack.common.rpc.amqp remote='dest')
2013-10-09 11:39:43.657 TRACE cinder.openstack.common.rpc.amqp File "/opt/stack/cinder/cinder/volume/driver.py", line 293, in copy_volume_data
2013-10-09 11:39:43.657 TRACE cinder.openstack.common.rpc.amqp {'status': dest_orig_status})
2013-10-09 11:39:43.657 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-10-09 11:39:43.657 TRACE cinder.openstack.common.rpc.amqp self.gen.next()
2013-10-09 11:39:43.657 TRACE cinder.openstack.common.rpc.amqp File "/opt/stack/cinder/cinder/volume/driver.py", line 287, in copy_volume_data
2013-10-09 11:39:43.657 TRACE cinder.openstack.common.rpc.amqp remote=dest_remote)
2013-10-09 11:39:43.657 TRACE cinder.openstack.common.rpc.amqp File "/opt/stack/cinder/cinder/volume/driver.py", line 378, in _attach_volume
2013-10-09 11:39:43.657 TRACE cinder.openstack.common.rpc.amqp device_scan_attempts)
2013-10-09 11:39:43.657 TRACE cinder.openstack.common.rpc.amqp File "/opt/stack/cinder/cinder/utils.py", line 798, in brick_get_connector
2013-10-09 11:39:43.657 TRACE cinder.openstack.common.rpc.amqp device_scan_attempts)
2013-10-09 11:39:43.657 TRACE cinder.openstack.common.rpc.amqp File "/opt/stack/cinder/cinder/brick/initiator/connector.py", line 114, in factory
2013-10-09 11:39:43.657 TRACE cinder.openstack.common.rpc.amqp *args, **kwargs)
2013-10-09 11:39:43.657 TRACE cinder.openstack.common.rpc.amqp File "/opt/stack/cinder/cinder/brick/initiator/connector.py", line 802, in __init__
2013-10-09 11:39:43.657 TRACE cinder.openstack.common.rpc.amqp *args, **kwargs)
2013-10-09 11:39:43.657 TRACE cinder.openstack.common.rpc.amqp File "/opt/stack/cinder/cinder/brick/remotefs/remotefs.py", line 41, in __init__
2013-10-09 11:39:43.657 TRACE cinder.openstack.common.rpc.amqp err=_('nfs_mount_point_base required'))
2013-10-09 11:39:43.657 TRACE cinder.openstack.common.rpc.amqp InvalidParameterValue: An unknown exception occurred.

[Zhenshi Zhou (joker-zhou)]

Hi Jeff,

Cinder pieces 'nfs_mount_point_base' and the hash string of device name together to get the specific mount point. It is necessary to provide this setting so that Cinder would know where the mount points are when it tries to attach/deattach the volumes.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/53316

[Navneet (singn)]

I guess this was also a problem while doing a back up and restore for NFS drivers even though I dont see that being reported. The patch should fix those problems.

[Eric Harney (eharney)]

Can you provide a little more info about how to trigger this? I'd like to be able to tell if the GlusterFS driver has this same problem. (Suspecting it might.)

[Navneet (singn)]

Looking at the connector code I did suspect that GlusterFS might also have the same problem but I was not fully aware of the specifics of the driver so did not mention it. The simplest way to see if your driver has the same problem is to configure multiple backends. Try migrating volume from src to dest backend using cinder migrate command. Any of the src or dest should be GlusterFS backend. If you support your backend for backup and restore then this might also affect those cases.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/53316
Committed: http://github.com/openstack/cinder/commit/b6b9df2940ac07f715f759f4c315b2cf088c2320
Submitter: Jenkins
Branch: master

commit b6b9df2940ac07f715f759f4c315b2cf088c2320
Author: Navneet Singh <email address hidden>
Date: Sun Oct 20 18:30:36 2013 +0530

    Brick connector fix for NFS drivers

    This change fixes the error that comes
    while getting brick connector and attaching volumes
    in case of NFS drivers in cinder. The attribute for
    mount point base was not passed to attach_volume method
    as the connector initialization logic is common for all
    types of protocols. It is fixed by populating
    the required parameter in the RemoteFsConnector
    for NFS drivers.

    Change-Id: I8601326b318f6f8c53a03610f1b4f2bfd14070ff
    Closes-Bug: #1238085

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/53784

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/54002

[Navneet (singn)]

The infrastructure code for GlusterFs driver is implemented in this patch. GlusterFs driver needs to implement the method _get_mount_point_base to fix it for their backend.

[Ben Swartzlander (bswartz)]

This change was accepted but not merged yet because the gate jobs are backed up as usual.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/54002
Committed: http://github.com/openstack/cinder/commit/1cd362630655798e5ff4c058b3f44862ae61c801
Submitter: Jenkins
Branch: master

commit 1cd362630655798e5ff4c058b3f44862ae61c801
Author: Navneet Singh <email address hidden>
Date: Sun Oct 20 18:30:36 2013 +0530

    Brick connector revised fix for NFS drivers

    This change fixes the error that comes
    while getting brick connector and attaching volumes
    in case of NFS drivers in cinder. The attribute for
    mount point base was not passed to attach_volume method
    as the connector initialization logic is common for all
    types of protocols. It is fixed by populating
    the required parameter in the RemoteFsConnector
    by extracting it from connection properties. The infrastructure
    for fixing this situation for remote fs drivers is implemented
    in this fix. The drivers need to override the method
    _get_mount_point_base in their implementation. NFS driver
    implements it. GlusterFs implementation of the method to follow
    in a separate patch. Hence marking it partial fix.

    Change-Id: Id61a437fca1870529fa8b85f7fc2f73eae665294
    Partial-Bug: #1238085

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/58693

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/58693
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=b15118a78da790b1f80e358458dd2ec8a4d89393
Submitter: Jenkins
Branch: stable/havana

commit b15118a78da790b1f80e358458dd2ec8a4d89393
Author: Navneet Singh <email address hidden>
Date: Sun Oct 20 18:30:36 2013 +0530

    Brick connector revised fix for NFS drivers

    This change fixes the error that comes
    while getting brick connector and attaching volumes
    in case of NFS drivers in cinder. The attribute for
    mount point base was not passed to attach_volume method
    as the connector initialization logic is common for all
    types of protocols. It is fixed by populating
    the required parameter in the RemoteFsConnector
    by extracting it from connection properties. The infrastructure
    for fixing this situation for remote fs drivers is implemented
    in this fix. The drivers need to override the method
    _get_mount_point_base in their implementation. NFS driver
    implements it. GlusterFs implementation of the method to follow
    in a separate patch. Hence marking it partial fix.

    Change-Id: Id61a437fca1870529fa8b85f7fc2f73eae665294
    Partial-Bug: #1238085
    (cherry picked from commit 1cd362630655798e5ff4c058b3f44862ae61c801)

[Eric Harney (eharney)]

I think this wasn't auto-tagged as Released because the last fix was a Partial-Bug fix. I'll create a new bug for the related GlusterFS changes.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/70404

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/70405

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/70416

[Eric Harney (eharney)]

Moving back to In Progress and targeting for I-3 since the submitted fix was only Partial-Bug and there is a second one to include here w/ Closes-Bug. Should get the launchpad state tracking back in line.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/70405
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=4edde1fc938be9679fbf9c209a7f28b457b1bef1
Submitter: Jenkins
Branch: master

commit 4edde1fc938be9679fbf9c209a7f28b457b1bef1
Author: Eric Harney <email address hidden>
Date: Fri Dec 20 11:30:51 2013 -0500

    Brick connector fix for GlusterFS

    Fixes an InvalidParameterValue error by passing the required
    information to Brick for mounting.

    See also: 1cd3626 Brick connector revised fix for NFS

    Bump driver version 1.1.1.

    Closes-Bug: 1238085

    Change-Id: Ie45889460dd4775a556d592161c98c24a56989c7

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/70416
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=f134e3becb978c138c33de411c47fd9afa41900c
Submitter: Jenkins
Branch: master

commit f134e3becb978c138c33de411c47fd9afa41900c
Author: Walter A. Boring IV <email address hidden>
Date: Fri Jan 31 11:33:17 2014 -0800

    Brick fix BrickException message formatting

    This patch fixes an issue when passing in a
    formatting key and value for formatted
    text messages for all Brick exceptions.

    Added unit tests for brick's exception

    Change-Id: I76dcfc6872ca20305d1844162dd0ae28c4bb565c
    Related-Bug: 1238085