C-vol crashes when driver class is changed after some volumes exist on that backend

[George Shuklin]

Cinder-volume traces after calling invalid mount command:
sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t nfs -o hard 10.13.77.8:3260,1 iqn.2010-10.org.openstack:volume-00e5b727-0e23-4c12-ae7e-2eeb575fd443 1 /var/lib/cinder/mnt/9068d7d4ebb7ab3aac1cde9c50bc1406

Trace:
2014-03-03 12:02:02.376 1561 DEBUG cinder.openstack.common.processutils [req-4e984358-3b07-4f78-bb98-2e29975e176c None None] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t nfs -o hard 10.13.77.8:3260,1 iqn.2010-10.org.openstack:volume-00e5b727-0e23-4c12-ae7e-2eeb575fd443 1 /var/lib/cinder/mnt/9068d7d4ebb7ab3aac1cde9c50bc1406 execute /usr/lib/python2.7/dist-packages/cinder/openstack/common/processutils.py:142
2014-03-03 12:02:02.415 1561 DEBUG cinder.openstack.common.processutils [req-4e984358-3b07-4f78-bb98-2e29975e176c None None] Result was 32 execute /usr/lib/python2.7/dist-packages/cinder/openstack/common/processutils.py:167
2014-03-03 12:02:02.416 1561 ERROR cinder.service [req-4e984358-3b07-4f78-bb98-2e29975e176c None None] Unhandled exception
2014-03-03 12:02:02.416 1561 TRACE cinder.service Traceback (most recent call last):
2014-03-03 12:02:02.416 1561 TRACE cinder.service File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 228, in _start_child
2014-03-03 12:02:02.416 1561 TRACE cinder.service self._child_process(wrap.server)
2014-03-03 12:02:02.416 1561 TRACE cinder.service File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 205, in _child_process
2014-03-03 12:02:02.416 1561 TRACE cinder.service launcher.run_server(server)
2014-03-03 12:02:02.416 1561 TRACE cinder.service File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 96, in run_server
2014-03-03 12:02:02.416 1561 TRACE cinder.service server.start()
2014-03-03 12:02:02.416 1561 TRACE cinder.service File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 385, in start
2014-03-03 12:02:02.416 1561 TRACE cinder.service self.manager.init_host()
2014-03-03 12:02:02.416 1561 TRACE cinder.service File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 209, in init_host
2014-03-03 12:02:02.416 1561 TRACE cinder.service self.driver.ensure_export(ctxt, volume)
2014-03-03 12:02:02.416 1561 TRACE cinder.service File "/usr/lib/python2.7/dist-packages/cinder/volume/drivers/nfs.py", line 167, in ensure_export
2014-03-03 12:02:02.416 1561 TRACE cinder.service self._ensure_share_mounted(volume['provider_location'])
2014-03-03 12:02:02.416 1561 TRACE cinder.service File "/usr/lib/python2.7/dist-packages/cinder/volume/drivers/nfs.py", line 435, in _ensure_share_mounted
2014-03-03 12:02:02.416 1561 TRACE cinder.service self._remotefsclient.mount(nfs_share, mnt_flags)
2014-03-03 12:02:02.416 1561 TRACE cinder.service File "/usr/lib/python2.7/dist-packages/cinder/brick/remotefs/remotefs.py", line 100, in mount
2014-03-03 12:02:02.416 1561 TRACE cinder.service run_as_root=True, check_exit_code=0)
2014-03-03 12:02:02.416 1561 TRACE cinder.service File "/usr/lib/python2.7/dist-packages/cinder/openstack/common/processutils.py", line 173, in execute
2014-03-03 12:02:02.416 1561 TRACE cinder.service cmd=' '.join(cmd))
2014-03-03 12:02:02.416 1561 TRACE cinder.service ProcessExecutionError: Unexpected error while running command.
2014-03-03 12:02:02.416 1561 TRACE cinder.service Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t nfs -o hard 78.140.137.211:3260,1 iqn.2010-10.org.openstack:volume-00e5b727-0e23-4c12-ae7e-2eeb575fd443 1 /var/lib/cinder/mnt/9068d7d4ebb7ab3aac1cde9c50bc1406
2014-03-03 12:02:02.416 1561 TRACE cinder.service Exit code: 32
2014-03-03 12:02:02.416 1561 TRACE cinder.service Stdout: ''
2014-03-03 12:02:02.416 1561 TRACE cinder.service Stderr: 'mount.nfs: access denied by server while mounting 78.140.137.211:3260,1 iqn.2010-10.org.openstack:volume-00e5b727-0e23-4c12-ae7e-2eeb575fd443 1\n'

content of cinder.conf:

[DEFAULT]
rabbit_host = srv
rabbit_port = 5672
rabbit_userid = guest
rabbit_password = pass
rootwrap_config = /etc/cinder/rootwrap.conf
api_paste_confg = /etc/cinder/api-paste.ini
iscsi_helper = tgtadm
volume_name_template = volume-%s
volume_group = cinder-volumes
verbose = True
auth_strategy = keystone
state_path = /var/lib/cinder
lock_path = /var/lock/cinder
volumes_dir = /var/lib/cinder/volumes
volume_driver=cinder.volume.drivers.nfs.NfsDriver
nfs_mount_options=hard
debug=True
verbose=True
[database]
connection = mysql://cinder:pass@srv/cinder

content of /etc/cinder/nfs_shares:
10.13.77.8:/storage

content of /etc/exports:
/storage *(rw,async)ii cinder-common 1:2013.2.1-0ubuntu1~cloud0 Cinder storage service - common files
ii cinder-volume 1:2013.2.1-0ubuntu1~cloud0 Cinder storage service - Volume server
ii python-cinder 1:2013.2.1-0ubuntu1~cloud0 Cinder Python libraries

10.13.77.8 - local IP of server where cinder-volume is running. Manually NFS share mounting without problems, cinder works fine with LVM backend.

Package information:
Ubuntu 12.04 + cloud archive.

ii cinder-common 1:2013.2.1-0ubuntu1~cloud0 Cinder storage service - common files
ii cinder-volume 1:2013.2.1-0ubuntu1~cloud0 Cinder storage service - Volume server
ii python-cinder 1:2013.2.1-0ubuntu1~cloud0 Cinder Python libraries

[Eric Harney (eharney)]

I believe this happens if you create a volume with the LVM iSCSI driver, then change cinder.conf to use the NFS driver and restart the volume service.

It tries to use the provider_location from the LVM driver's volume and results in this.

[George Shuklin (george-shuklin)]

Yes, it is. But I think this error should be handled somehow differ (at least without attemt to 'mount' IQN).

[Ben Swartzlander (bswartz)]

I agree with both of the above comments. The NFS driver could probably check the provider_location and determine that that volume was created by a different driver and return a more sane error message. However this seems like a low priority because the user clearly did something that will not succeed.