Backup to NFS share failed: incorrent mount options

[yangbo]

2015-12-27 21:21:44.807 DEBUG os_brick.remotefs.remotefs [req-0680b554-47e4-4a99-abee-042268826751 admin] Failed to do nfs mount. _mount_nfs /usr/local/lib/python2.7/dist-packages/os_brick/remotefs/remotefs.py:146
2015-12-27 21:21:44.890 ERROR oslo_messaging.rpc.dispatcher [req-0680b554-47e4-4a99-abee-042268826751 admin] Exception during message handling: NFS mount failed for share 10.109.196.253:/BackupFS2. Error - {'pnfs': u"Unexpected error while running command.\nCommand: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t nfs -o None,vers=4,minorversion=1 10.109.196.253:/BackupFS2 /opt/stack/data/cinder/backup_mount/b83c6af067d5e837c7c545b3f4e629f1\nExit code: 32\nStdout: u''\nStderr: u'mount.nfs: an incorrect mount option was specified\\n'", 'nfs': u"Unexpected error while running command.\nCommand: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t nfs -o None 10.109.196.253:/BackupFS2 /opt/stack/data/cinder/backup_mount/b83c6af067d5e837c7c545b3f4e629f1\nExit code: 32\nStdout: u''\nStderr: u'mount.nfs: an incorrect mount option was specified\\n'"}
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher executor_callback))
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher executor_callback)
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/backup/manager.py", line 408, in create_backup
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher self._update_backup_error(backup, context, six.text_type(err))
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/backup/manager.py", line 400, in create_backup
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher backup_service = self.service.get_backup_driver(context)
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/backup/drivers/nfs.py", line 80, in get_backup_driver
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher return NFSBackupDriver(context)
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/backup/drivers/nfs.py", line 55, in __init__
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher backup_path = self._init_backup_repo_path()
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/backup/drivers/nfs.py", line 75, in _init_backup_repo_path
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher remotefsclient.mount(self.backup_share)
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/os_brick/remotefs/remotefs.py", line 110, in mount
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher self._mount_nfs(share, mount_path, flags)
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/os_brick/remotefs/remotefs.py", line 150, in _mount_nfs
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher 'error': mnt_errors})
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher BrickException: NFS mount failed for share 10.109.196.253:/BackupFS2. Error - {'pnfs': u"Unexpected error while running command.\nCommand: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t nfs -o None,vers=4,minorversion=1 10.109.196.253:/BackupFS2 /opt/stack/data/cinder/backup_mount/b83c6af067d5e837c7c545b3f4e629f1\nExit code: 32\nStdout: u''\nStderr: u'mount.nfs: an incorrect mount option was specified\\n'", 'nfs': u"Unexpected error while running command.\nCommand: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t nfs -o None 10.109.196.253:/BackupFS2 /opt/stack/data/cinder/backup_mount/b83c6af067d5e837c7c545b3f4e629f1\nExit code: 32\nStdout: u''\nStderr: u'mount.nfs: an incorrect mount option was specified\\n'"}

Openstack is installed on Ubuntu Server.

I tried to remove the options then it succeeded, so the options are incorrect , should be modified.

stack@compute1:/opt/stack/logs/screen$ sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t nfs -o None,vers=4,minorversion=1 10.109.196.253:/BackupFS2 /opt/stack/data/cinder/backup_mount/b83c6af067d5e837c7c545b3f4e629f1
mount.nfs: an incorrect mount option was specified

stack@compute1:/opt/stack/logs/screen$ sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t nfs 10.109.196.253:/BackupFS2 /opt/stack/data/cinder/backup_mount/b83c6af067d5e837c7c545b3f4e629f1
stack@compute1:/opt/stack/logs/screen$ mount |grep Backup
10.109.196.253:/BackupFS2 on /opt/stack/data/cinder/backup_mount/b83c6af067d5e837c7c545b3f4e629f1 type nfs (rw,vers=4,addr=10.109.196.253,clientaddr=10.109.194.147)

[yangbo (yangbo-jiang)]

Hi Sharma,

Any comment?

[Eric Harney (eharney)]

Can you post your cinder.conf settings?

[yangbo (yangbo-jiang)]

backup_driver = cinder.backup.drivers.nfs
backup_container = None
backup_enable_progress_timer = True
backup_file_size = 1999994880
backup_mount_options = None
backup_mount_point_base = $state_path/backup_mount
backup_sha_block_size_bytes = 32768
backup_share =10.109.196.253:/BackupFS2
[database]
max_pool_size = 60
max_overflow = 90
connection = mysql+pymysql://root:stack@127.0.0.1/cinder?charset=utf8

[oslo_concurrency]
lock_path = /opt/stack/data/cinder

[lvmdriver-1]
lvm_type = default
iscsi_helper = tgtadm
volume_group = stack-volumes-lvmdriver-1
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
volume_backend_name = lvmdriver-1

[oslo_messaging_rabbit]
rabbit_userid = stackrabbit
rabbit_password = stack
rabbit_hosts = 10.109.194.142

[JF-D1677]
volume_backend_name = JF-D1677
storage_vnx_pool_name = OpenStack_Test_01
san_ip = 10.109.196.48
san_secondary_ip = 10.109.196.49
san_login = sysadmin
san_password = sysadmin
storage_vnx_authentication_type = global
naviseccli_path = /opt/Navisphere/bin/naviseccli
default_timeout = 30
volume_driver=cinder.volume.drivers.emc.emc_cli_iscsi.EMCCLIISCSIDriver
destroy_empty_storage_group = False
iscsi_initiators={"compute1":["10.109.196.250","10.109.196.252","10.109.196.251"]}
max_luns_per_storage_pool = 1000
initiator_auto_registration = True

[Eric Harney (eharney)]

You probably should not set

 backup_mount_options = None

But, this does look like something we should clean up.

[yangbo (yangbo-jiang)]

I removed some options, and still the same issue.

2016-01-17 19:27:48.038 ERROR oslo_messaging.rpc.dispatcher [req-34870075-51de-4ccb-8ba1-64f07f9711ed admin] Exception during message handling: NFS mount failed for share 10.109.196.253:/BackupFS2. Error - {'pnfs': u"Unexpected error while running command.\nCommand: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t nfs -o vers=4,minorversion=1 10.109.196.253:/BackupFS2 /opt/stack/data/cinder/backup_mount/b83c6af067d5e837c7c545b3f4e629f1\nExit code: 32\nStdout: u''\nStderr: u'mount.nfs: Operation not permitted\\n'", 'nfs': u"Unexpected error while running command.\nCommand: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t nfs -o {} 10.109.196.253:/BackupFS2 /opt/stack/data/cinder/backup_mount/b83c6af067d5e837c7c545b3f4e629f1\nExit code: 32\nStdout: u''\nStderr: u'mount.nfs: an incorrect mount option was specified\\n'"}
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispatcher executor_callback))
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispatcher executor_callback)
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/backup/manager.py", line 408, in create_backup
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispatcher self._update_backup_error(backup, context, six.text_type(err))
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/backup/manager.py", line 400, in create_backup
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispatcher backup_service = self.service.get_backup_driver(context)
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/backup/drivers/nfs.py", line 80, in get_backup_driver
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispatcher return NFSBackupDriver(context)
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispa...

[Xiangfei Zhu (xiangfeiz)]

I just met a similar issue and it turns out that cinder backup driver is expecting nfs version 4.1 by default. I guess your nfs version is not 4.1. You can set backup_mount_options to fit your nfs version.

For nfs 3, set backup_mount_options="vers=3"
For nfs 4.0, set backup_mount_options="vers=4,minorversion=0"

I hope it helps.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/351890

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/352299

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/352300

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ivan Kolodyazhny (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/352299

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ivan Kolodyazhny (<email address hidden>) on branch: stable/liberty
Review: https://review.openstack.org/352300

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/351890
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=ca25d7af9768182723a64fd26be3a8a41142c001
Submitter: Jenkins
Branch: master

commit ca25d7af9768182723a64fd26be3a8a41142c001
Author: Ivan Kolodyazhny <email address hidden>
Date: Fri Aug 5 22:23:34 2016 +0300

    Fix backup NFS share mount with default backup_mount_options

    Now, if backup_mount_options in not set, NFS Backup driver
    passes empty dict to RemoteFsClient. It cause an error when
    RemoteFsClient mounts the share because it passes '-o {}' params.

    According to [1] we should pass None to RemoteFsClient if no
    backup_mount_options was specified.

    [1] https://github.com/openstack/os-brick/blob/master/os_brick/remotefs/remotefs.py#L129-L130

    Change-Id: Ief874332a71ff77c88a65e2d631457b831472236
    Closes-Bug: #1529566

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/387642

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/387642
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=5b387121f1d883e08bf439ba856fe17eb77017cc
Submitter: Jenkins
Branch: stable/newton

commit 5b387121f1d883e08bf439ba856fe17eb77017cc
Author: Ivan Kolodyazhny <email address hidden>
Date: Fri Aug 5 22:23:34 2016 +0300

    Fix backup NFS share mount with default backup_mount_options

    Now, if backup_mount_options in not set, NFS Backup driver
    passes empty dict to RemoteFsClient. It cause an error when
    RemoteFsClient mounts the share because it passes '-o {}' params.

    According to [1] we should pass None to RemoteFsClient if no
    backup_mount_options was specified.

    [1] https://github.com/openstack/os-brick/blob/master/os_brick/remotefs/remotefs.py#L129-L130

    Change-Id: Ief874332a71ff77c88a65e2d631457b831472236
    Closes-Bug: #1529566
    (cherry picked from commit ca25d7af9768182723a64fd26be3a8a41142c001)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 10.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 9.1.1 release.