Backup to NFS share failed: incorrent mount options

Bug #1529566 reported by yangbo
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Low
Ivan Kolodyazhny

Bug Description

2015-12-27 21:21:44.807 DEBUG os_brick.remotefs.remotefs [req-0680b554-47e4-4a99-abee-042268826751 admin] Failed to do nfs mount. _mount_nfs /usr/local/lib/python2.7/dist-packages/os_brick/remotefs/remotefs.py:146
2015-12-27 21:21:44.890 ERROR oslo_messaging.rpc.dispatcher [req-0680b554-47e4-4a99-abee-042268826751 admin] Exception during message handling: NFS mount failed for share 10.109.196.253:/BackupFS2. Error - {'pnfs': u"Unexpected error while running command.\nCommand: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t nfs -o None,vers=4,minorversion=1 10.109.196.253:/BackupFS2 /opt/stack/data/cinder/backup_mount/b83c6af067d5e837c7c545b3f4e629f1\nExit code: 32\nStdout: u''\nStderr: u'mount.nfs: an incorrect mount option was specified\\n'", 'nfs': u"Unexpected error while running command.\nCommand: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t nfs -o None 10.109.196.253:/BackupFS2 /opt/stack/data/cinder/backup_mount/b83c6af067d5e837c7c545b3f4e629f1\nExit code: 32\nStdout: u''\nStderr: u'mount.nfs: an incorrect mount option was specified\\n'"}
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher executor_callback))
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher executor_callback)
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/backup/manager.py", line 408, in create_backup
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher self._update_backup_error(backup, context, six.text_type(err))
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/backup/manager.py", line 400, in create_backup
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher backup_service = self.service.get_backup_driver(context)
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/backup/drivers/nfs.py", line 80, in get_backup_driver
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher return NFSBackupDriver(context)
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/backup/drivers/nfs.py", line 55, in __init__
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher backup_path = self._init_backup_repo_path()
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/backup/drivers/nfs.py", line 75, in _init_backup_repo_path
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher remotefsclient.mount(self.backup_share)
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/os_brick/remotefs/remotefs.py", line 110, in mount
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher self._mount_nfs(share, mount_path, flags)
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/os_brick/remotefs/remotefs.py", line 150, in _mount_nfs
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher 'error': mnt_errors})
2015-12-27 21:21:44.890 1165 ERROR oslo_messaging.rpc.dispatcher BrickException: NFS mount failed for share 10.109.196.253:/BackupFS2. Error - {'pnfs': u"Unexpected error while running command.\nCommand: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t nfs -o None,vers=4,minorversion=1 10.109.196.253:/BackupFS2 /opt/stack/data/cinder/backup_mount/b83c6af067d5e837c7c545b3f4e629f1\nExit code: 32\nStdout: u''\nStderr: u'mount.nfs: an incorrect mount option was specified\\n'", 'nfs': u"Unexpected error while running command.\nCommand: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t nfs -o None 10.109.196.253:/BackupFS2 /opt/stack/data/cinder/backup_mount/b83c6af067d5e837c7c545b3f4e629f1\nExit code: 32\nStdout: u''\nStderr: u'mount.nfs: an incorrect mount option was specified\\n'"}

Openstack is installed on Ubuntu Server.

I tried to remove the options then it succeeded, so the options are incorrect , should be modified.

stack@compute1:/opt/stack/logs/screen$ sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t nfs -o None,vers=4,minorversion=1 10.109.196.253:/BackupFS2 /opt/stack/data/cinder/backup_mount/b83c6af067d5e837c7c545b3f4e629f1
mount.nfs: an incorrect mount option was specified

stack@compute1:/opt/stack/logs/screen$ sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t nfs 10.109.196.253:/BackupFS2 /opt/stack/data/cinder/backup_mount/b83c6af067d5e837c7c545b3f4e629f1
stack@compute1:/opt/stack/logs/screen$ mount |grep Backup
10.109.196.253:/BackupFS2 on /opt/stack/data/cinder/backup_mount/b83c6af067d5e837c7c545b3f4e629f1 type nfs (rw,vers=4,addr=10.109.196.253,clientaddr=10.109.194.147)

aditi sharma (adi-sky17)
Changed in cinder:
assignee: nobody → aditi sharma (adi-sky17)
Revision history for this message
yangbo (yangbo-jiang) wrote :

Hi Sharma,

Any comment?

Revision history for this message
Eric Harney (eharney) wrote :

Can you post your cinder.conf settings?

Revision history for this message
yangbo (yangbo-jiang) wrote :

backup_driver = cinder.backup.drivers.nfs
backup_container = None
backup_enable_progress_timer = True
backup_file_size = 1999994880
backup_mount_options = None
backup_mount_point_base = $state_path/backup_mount
backup_sha_block_size_bytes = 32768
backup_share =10.109.196.253:/BackupFS2
[database]
max_pool_size = 60
max_overflow = 90
connection = mysql+pymysql://root:stack@127.0.0.1/cinder?charset=utf8

[oslo_concurrency]
lock_path = /opt/stack/data/cinder

[lvmdriver-1]
lvm_type = default
iscsi_helper = tgtadm
volume_group = stack-volumes-lvmdriver-1
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
volume_backend_name = lvmdriver-1

[oslo_messaging_rabbit]
rabbit_userid = stackrabbit
rabbit_password = stack
rabbit_hosts = 10.109.194.142

[JF-D1677]
volume_backend_name = JF-D1677
storage_vnx_pool_name = OpenStack_Test_01
san_ip = 10.109.196.48
san_secondary_ip = 10.109.196.49
san_login = sysadmin
san_password = sysadmin
storage_vnx_authentication_type = global
naviseccli_path = /opt/Navisphere/bin/naviseccli
default_timeout = 30
volume_driver=cinder.volume.drivers.emc.emc_cli_iscsi.EMCCLIISCSIDriver
destroy_empty_storage_group = False
iscsi_initiators={"compute1":["10.109.196.250","10.109.196.252","10.109.196.251"]}
max_luns_per_storage_pool = 1000
initiator_auto_registration = True

Revision history for this message
Eric Harney (eharney) wrote :

You probably should not set

 backup_mount_options = None

But, this does look like something we should clean up.

Changed in cinder:
importance: Undecided → Low
Revision history for this message
yangbo (yangbo-jiang) wrote :
Download full text (5.3 KiB)

I removed some options, and still the same issue.

2016-01-17 19:27:48.038 ERROR oslo_messaging.rpc.dispatcher [req-34870075-51de-4ccb-8ba1-64f07f9711ed admin] Exception during message handling: NFS mount failed for share 10.109.196.253:/BackupFS2. Error - {'pnfs': u"Unexpected error while running command.\nCommand: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t nfs -o vers=4,minorversion=1 10.109.196.253:/BackupFS2 /opt/stack/data/cinder/backup_mount/b83c6af067d5e837c7c545b3f4e629f1\nExit code: 32\nStdout: u''\nStderr: u'mount.nfs: Operation not permitted\\n'", 'nfs': u"Unexpected error while running command.\nCommand: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t nfs -o {} 10.109.196.253:/BackupFS2 /opt/stack/data/cinder/backup_mount/b83c6af067d5e837c7c545b3f4e629f1\nExit code: 32\nStdout: u''\nStderr: u'mount.nfs: an incorrect mount option was specified\\n'"}
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispatcher executor_callback))
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispatcher executor_callback)
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/backup/manager.py", line 408, in create_backup
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispatcher self._update_backup_error(backup, context, six.text_type(err))
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/backup/manager.py", line 400, in create_backup
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispatcher backup_service = self.service.get_backup_driver(context)
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/backup/drivers/nfs.py", line 80, in get_backup_driver
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispatcher return NFSBackupDriver(context)
2016-01-17 19:27:48.038 31192 ERROR oslo_messaging.rpc.dispa...

Read more...

aditi sharma (adi-sky17)
Changed in cinder:
assignee: aditi sharma (adi-sky17) → nobody
aditi sharma (adi-sky17)
Changed in cinder:
assignee: nobody → aditi sharma (adi-sky17)
Revision history for this message
Xiangfei Zhu (xiangfeiz) wrote :

I just met a similar issue and it turns out that cinder backup driver is expecting nfs version 4.1 by default. I guess your nfs version is not 4.1. You can set backup_mount_options to fit your nfs version.

For nfs 3, set backup_mount_options="vers=3"
For nfs 4.0, set backup_mount_options="vers=4,minorversion=0"

I hope it helps.

Ivan Kolodyazhny (e0ne)
Changed in cinder:
status: New → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/351890

Changed in cinder:
assignee: aditi sharma (adi-sky17) → Ivan Kolodyazhny (e0ne)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/352299

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/352300

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on cinder (stable/mitaka)

Change abandoned by Ivan Kolodyazhny (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/352299

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on cinder (stable/liberty)

Change abandoned by Ivan Kolodyazhny (<email address hidden>) on branch: stable/liberty
Review: https://review.openstack.org/352300

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/351890
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=ca25d7af9768182723a64fd26be3a8a41142c001
Submitter: Jenkins
Branch: master

commit ca25d7af9768182723a64fd26be3a8a41142c001
Author: Ivan Kolodyazhny <email address hidden>
Date: Fri Aug 5 22:23:34 2016 +0300

    Fix backup NFS share mount with default backup_mount_options

    Now, if backup_mount_options in not set, NFS Backup driver
    passes empty dict to RemoteFsClient. It cause an error when
    RemoteFsClient mounts the share because it passes '-o {}' params.

    According to [1] we should pass None to RemoteFsClient if no
    backup_mount_options was specified.

    [1] https://github.com/openstack/os-brick/blob/master/os_brick/remotefs/remotefs.py#L129-L130

    Change-Id: Ief874332a71ff77c88a65e2d631457b831472236
    Closes-Bug: #1529566

Changed in cinder:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/387642

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/newton)

Reviewed: https://review.openstack.org/387642
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=5b387121f1d883e08bf439ba856fe17eb77017cc
Submitter: Jenkins
Branch: stable/newton

commit 5b387121f1d883e08bf439ba856fe17eb77017cc
Author: Ivan Kolodyazhny <email address hidden>
Date: Fri Aug 5 22:23:34 2016 +0300

    Fix backup NFS share mount with default backup_mount_options

    Now, if backup_mount_options in not set, NFS Backup driver
    passes empty dict to RemoteFsClient. It cause an error when
    RemoteFsClient mounts the share because it passes '-o {}' params.

    According to [1] we should pass None to RemoteFsClient if no
    backup_mount_options was specified.

    [1] https://github.com/openstack/os-brick/blob/master/os_brick/remotefs/remotefs.py#L129-L130

    Change-Id: Ief874332a71ff77c88a65e2d631457b831472236
    Closes-Bug: #1529566
    (cherry picked from commit ca25d7af9768182723a64fd26be3a8a41142c001)

tags: added: in-stable-newton
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 10.0.0.0b1

This issue was fixed in the openstack/cinder 10.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 9.1.1

This issue was fixed in the openstack/cinder 9.1.1 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.