Command to extend volume fails using generic NFS driver. After executing it the volume is in status 'error_extending'. The problem is in the qemu-img can't lock for write, since other proccess is using (as log shows). The error occours only while the volume is attached on an Nova instance, so if I run the command on an volume without it being attached, it works.
# Error log (cinder-volume service)
Apr 02 13:46:01 47-throne-dev-openstack cinder-volume[27508]: ERROR cinder.volume.manager [req-f190705c-c270-4280-9c13-31eab0580797 req-971d7a69-ae16-4049-9ea2-36051f2
0535d admin None] Extend volume failed.: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Apr 02 13:46:01 47-throne-dev-openstack cinder-volume[27508]: Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf qemu-img resize /opt/stack/data/cinder/mnt/1af9e5
980aa9bae6e080782e4ebad95e/volume-613d3202-052c-4c4d-b0eb-62117a9de081 2G
Apr 02 13:46:01 47-throne-dev-openstack cinder-volume[27508]: Exit code: 1
Apr 02 13:46:01 47-throne-dev-openstack cinder-volume[27508]: Stdout: ''
Apr 02 13:46:01 47-throne-dev-openstack cinder-volume[27508]: Stderr: 'WARNING: Image format was not specified for \'/opt/stack/data/cinder/mnt/1af9e5980aa9bae6e080782e4ebad95e/volume-613d3202-052c-4c4d-b0eb-62117a9de081\' and probing guessed raw.\n Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.\n Specify the \'raw\' format explicitly to remove the restrictions.\nqemu-img: Could not open \'/opt/stack/data/cinder/mnt/1af9e5980aa9bae6e080782e4ebad95e/volume-613d3202-052c-4c4d-b0eb-62117a9de081\': Failed to get "write" lock\nIs another process using the image?\n'
Apr 02 13:46:01 47-throne-dev-openstack cinder-volume[27508]: ERROR cinder.volume.manager Traceback (most recent call last):
Apr 02 13:46:01 47-throne-dev-openstack cinder-volume[27508]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/manager.py", line 2740, in extend_volume
Apr 02 13:46:01 47-throne-dev-openstack cinder-volume[27508]: ERROR cinder.volume.manager self.driver.extend_volume(volume, new_size)
Apr 02 13:46:01 47-throne-dev-openstack cinder-volume[27508]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/nfs.py", line 376, in extend_volume
Apr 02 13:46:01 47-throne-dev-openstack cinder-volume[27508]: ERROR cinder.volume.manager run_as_root=self._execute_as_root)
Apr 02 13:46:01 47-throne-dev-openstack cinder-volume[27508]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/image/image_utils.py", line 354, in resize_image
Apr 02 13:46:01 47-throne-dev-openstack cinder-volume[27508]: ERROR cinder.volume.manager utils.execute(*cmd, run_as_root=run_as_root)
Apr 02 13:46:01 47-throne-dev-openstack cinder-volume[27508]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/utils.py", line 126, in execute
Apr 02 13:46:01 47-throne-dev-openstack cinder-volume[27508]: ERROR cinder.volume.manager return processutils.execute(*cmd, **kwargs)
Apr 02 13:46:01 47-throne-dev-openstack cinder-volume[27508]: ERROR cinder.volume.manager File "/usr/local/lib/python3.6/dist-packages/oslo_concurrency/processutils.py", line 424, in execute
Apr 02 13:46:01 47-throne-dev-openstack cinder-volume[27508]: ERROR cinder.volume.manager cmd=sanitized_cmd)
Apr 02 13:46:01 47-throne-dev-openstack cinder-volume[27508]: ERROR cinder.volume.manager oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Apr 02 13:46:01 47-throne-dev-openstack cinder-volume[27508]: ERROR cinder.volume.manager Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf qemu-img resize /opt/stack/data/cinder/mnt/1af9e5980aa9bae6e080782e4ebad95e/volume-613d3202-052c-4c4d-b0eb-62117a9de081 2G
Apr 02 13:46:01 47-throne-dev-openstack cinder-volume[27508]: ERROR cinder.volume.manager Exit code: 1
Apr 02 13:46:01 47-throne-dev-openstack cinder-volume[27508]: ERROR cinder.volume.manager Stdout: ''
Apr 02 13:46:01 47-throne-dev-openstack cinder-volume[27508]: ERROR cinder.volume.manager Stderr: 'WARNING: Image format was not specified for \'/opt/stack/data/cinder/mnt/1af9e5980aa9bae6e080782e4ebad95e/volume-613d3202-052c-4c4d-b0eb-62117a9de081\' and probing guessed raw.\n Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.\n Specify the \'raw\' format explicitly to remove the restrictions.\nqemu-img: Could not open \'/opt/stack/data/cinder/mnt/1af9e5980aa9bae6e080782e4ebad95e/volume-613d3202-052c-4c4d-b0eb-62117a9de081\': Failed to get "write" lock\nIs another process using the image?\n'
Apr 02 13:46:01 47-throne-dev-openstack cinder-volume[27508]: ERROR cinder.volume.manager
Command to extend volume fails using generic NFS driver. After executing it the volume is in status 'error_extending'. The problem is in the qemu-img can't lock for write, since other proccess is using (as log shows). The error occours only while the volume is attached on an Nova instance, so if I run the command on an volume without it being attached, it works.
# Error log (cinder-volume service)
Apr 02 13:46:01 47-throne- dev-openstack cinder- volume[ 27508]: ERROR cinder. volume. manager [req-f190705c- c270-4280- 9c13-31eab05807 97 req-971d7a69- ae16-4049- 9ea2-36051f2 y.processutils. ProcessExecutio nError: Unexpected error while running command. dev-openstack cinder- volume[ 27508]: Command: sudo cinder-rootwrap /etc/cinder/ rootwrap. conf qemu-img resize /opt/stack/ data/cinder/ mnt/1af9e5 82e4ebad95e/ volume- 613d3202- 052c-4c4d- b0eb-62117a9de0 81 2G dev-openstack cinder- volume[ 27508]: Exit code: 1 dev-openstack cinder- volume[ 27508]: Stdout: '' dev-openstack cinder- volume[ 27508]: Stderr: 'WARNING: Image format was not specified for \'/opt/ stack/data/ cinder/ mnt/1af9e5980aa 9bae6e080782e4e bad95e/ volume- 613d3202- 052c-4c4d- b0eb-62117a9de0 81\' and probing guessed raw.\n Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.\n Specify the \'raw\' format explicitly to remove the restrictions. \nqemu- img: Could not open \'/opt/ stack/data/ cinder/ mnt/1af9e5980aa 9bae6e080782e4e bad95e/ volume- 613d3202- 052c-4c4d- b0eb-62117a9de0 81\': Failed to get "write" lock\nIs another process using the image?\n' dev-openstack cinder- volume[ 27508]: ERROR cinder. volume. manager Traceback (most recent call last): dev-openstack cinder- volume[ 27508]: ERROR cinder. volume. manager File "/opt/stack/ cinder/ cinder/ volume/ manager. py", line 2740, in extend_volume dev-openstack cinder- volume[ 27508]: ERROR cinder. volume. manager self.driver. extend_ volume( volume, new_size) dev-openstack cinder- volume[ 27508]: ERROR cinder. volume. manager File "/opt/stack/ cinder/ cinder/ volume/ drivers/ nfs.py" , line 376, in extend_volume dev-openstack cinder- volume[ 27508]: ERROR cinder. volume. manager run_as_ root=self. _execute_ as_root) dev-openstack cinder- volume[ 27508]: ERROR cinder. volume. manager File "/opt/stack/ cinder/ cinder/ image/image_ utils.py" , line 354, in resize_image dev-openstack cinder- volume[ 27508]: ERROR cinder. volume. manager utils.execute(*cmd, run_as_ root=run_ as_root) dev-openstack cinder- volume[ 27508]: ERROR cinder. volume. manager File "/opt/stack/ cinder/ cinder/ utils.py" , line 126, in execute dev-openstack cinder- volume[ 27508]: ERROR cinder. volume. manager return processutils. execute( *cmd, **kwargs) dev-openstack cinder- volume[ 27508]: ERROR cinder. volume. manager File "/usr/local/ lib/python3. 6/dist- packages/ oslo_concurrenc y/processutils. py", line 424, in execute dev-openstack cinder- volume[ 27508]: ERROR cinder. volume. manager cmd=sanitized_cmd) dev-openstack cinder- volume[ 27508]: ERROR cinder. volume. manager oslo_concurrenc y.processutils. ProcessExecutio nError: Unexpected error while running command. dev-openstack cinder- volume[ 27508]: ERROR cinder. volume. manager Command: sudo cinder-rootwrap /etc/cinder/ rootwrap. conf qemu-img resize /opt/stack/ data/cinder/ mnt/1af9e5980aa 9bae6e080782e4e bad95e/ volume- 613d3202- 052c-4c4d- b0eb-62117a9de0 81 2G dev-openstack cinder- volume[ 27508]: ERROR cinder. volume. manager Exit code: 1 dev-openstack cinder- volume[ 27508]: ERROR cinder. volume. manager Stdout: '' dev-openstack cinder- volume[ 27508]: ERROR cinder. volume. manager Stderr: 'WARNING: Image format was not specified for \'/opt/ stack/data/ cinder/ mnt/1af9e5980aa 9bae6e080782e4e bad95e/ volume- 613d3202- 052c-4c4d- b0eb-62117a9de0 81\' and probing guessed raw.\n Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.\n Specify the \'raw\' format explicitly to remove the restrictions. \nqemu- img: Could not open \'/opt/ stack/data/ cinder/ mnt/1af9e5980aa 9bae6e080782e4e bad95e/ volume- 613d3202- 052c-4c4d- b0eb-62117a9de0 81\': Failed to get "write" lock\nIs another process using the image?\n' dev-openstack cinder- volume[ 27508]: ERROR cinder. volume. manager
0535d admin None] Extend volume failed.: oslo_concurrenc
Apr 02 13:46:01 47-throne-
980aa9bae6e0807
Apr 02 13:46:01 47-throne-
Apr 02 13:46:01 47-throne-
Apr 02 13:46:01 47-throne-
Apr 02 13:46:01 47-throne-
Apr 02 13:46:01 47-throne-
Apr 02 13:46:01 47-throne-
Apr 02 13:46:01 47-throne-
Apr 02 13:46:01 47-throne-
Apr 02 13:46:01 47-throne-
Apr 02 13:46:01 47-throne-
Apr 02 13:46:01 47-throne-
Apr 02 13:46:01 47-throne-
Apr 02 13:46:01 47-throne-
Apr 02 13:46:01 47-throne-
Apr 02 13:46:01 47-throne-
Apr 02 13:46:01 47-throne-
Apr 02 13:46:01 47-throne-
Apr 02 13:46:01 47-throne-
Apr 02 13:46:01 47-throne-
Apr 02 13:46:01 47-throne-
# Environment
-Devstack on Master Branch
-Ubuntu 18.04
- cinder.conf nfs_share volume. drivers. nfs.NfsDriver
[nfs]
nfs_shares_config = /etc/cinder/
volume_driver = cinder.
volume_backend_name = nfsbackend
- /etc/cinder/ nfs_share /mnt/sharedfold er
localhost:
- /etc/exports no_subtree_ check,no_ root_squash)
/mnt/sharedfolder *(rw,sync,