NFS: Creating multiple volumes from the same image fails when image caching is turned on by default

[Matt Riedemann]

Devstack recently changed to enable the cinder image cache by default:

https://github.com/openstack-dev/devstack/commit/dddb2c7b5f85688de9c9b92f025df25d2f2d3016

This means that after you create a volume from an image in the cache at least once, it tries to clone the volume on subsequent creates, and cloning isn't supported by the NFS driver.

So you end up with failures in cinder-volume like this:

http://logs.openstack.org/87/395887/4/check/gate-tempest-dsvm-full-devstack-plugin-nfs-nv/817e301/logs/screen-c-vol.txt.gz#_2016-11-13_15_29_26_733

2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager Traceback (most recent call last):
2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager File "/usr/local/lib/python2.7/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager result = task.execute(**arguments)
2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager File "/opt/stack/new/cinder/cinder/volume/flows/manager/create_volume.py", line 839, in execute
2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager **volume_spec)
2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager File "/opt/stack/new/cinder/cinder/volume/flows/manager/create_volume.py", line 788, in _create_from_image
2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager image_meta)
2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager File "/opt/stack/new/cinder/cinder/volume/manager.py", line 1134, in _create_image_cache_volume_entry
2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager image_meta)
2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager File "/opt/stack/new/cinder/cinder/volume/manager.py", line 1185, in _clone_image_volume
2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager self.create_volume(ctx, image_volume, allow_reschedule=False)
2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager File "<decorator-gen-235>", line 2, in create_volume
2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager File "/opt/stack/new/cinder/cinder/objects/cleanable.py", line 191, in wrapper
2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager result = f(*args, **kwargs)
2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager File "/opt/stack/new/cinder/cinder/volume/manager.py", line 610, in create_volume
2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager _run_flow()
2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager File "/opt/stack/new/cinder/cinder/volume/manager.py", line 602, in _run_flow
2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager flow_engine.run()
2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager File "/usr/local/lib/python2.7/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run
2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager for _state in self.run_iter(timeout=timeout):
2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager File "/usr/local/lib/python2.7/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager failure.Failure.reraise_if_any(er_failures)
2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager File "/usr/local/lib/python2.7/dist-packages/taskflow/types/failure.py", line 336, in reraise_if_any
2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager failures[0].reraise()
2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager File "/usr/local/lib/python2.7/dist-packages/taskflow/types/failure.py", line 343, in reraise
2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager six.reraise(*self._exc_info)
2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager File "/usr/local/lib/python2.7/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager result = task.execute(**arguments)
2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager File "/opt/stack/new/cinder/cinder/volume/flows/manager/create_volume.py", line 832, in execute
2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager context, volume, **volume_spec)
2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager File "/opt/stack/new/cinder/cinder/volume/flows/manager/create_volume.py", line 478, in _create_from_source_volume
2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager model_update = self.driver.create_cloned_volume(volume, srcvol_ref)
2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager File "/opt/stack/new/cinder/cinder/volume/driver.py", line 1515, in create_cloned_volume
2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager raise NotImplementedError()
2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager NotImplementedError
2016-11-13 15:29:26.733 825 ERROR cinder.volume.manager

The code here handles a CinderException trying to create the image cache from a volume entry:

https://github.com/openstack/cinder/blob/0408797d54d7db5fa3fbc936ae7b1d56c8adbe2b/cinder/volume/manager.py#L1174

Which fails from _clone_image_volume:

https://github.com/openstack/cinder/blob/0408797d54d7db5fa3fbc936ae7b1d56c8adbe2b/cinder/volume/manager.py#L1180

Because the NFS driver doesn't extend the CloneableImageVD mixin and implement the clone_image method.

So the volume manager code should probably be checking if the volume driver is an instance of CloneableImageVD before it tries going down this route.

The quick fix for the devstack-plugin-nfs code is to disable the image cache but this should still probably be fixed in cinder to not fail - or attempt an optional operation that's not supported by the driver.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/398581

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/398581
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=80b93f24b2e71a25594808fb2c41e7e7ce24b697
Submitter: Jenkins
Branch: master

commit 80b93f24b2e71a25594808fb2c41e7e7ce24b697
Author: Sean McGinnis <email address hidden>
Date: Wed Nov 16 15:15:30 2016 -0600

    Handle NotImplementedError for image cache cloning

    The base driver throws NotImplementedError, so if a driver does
    not override create_cloned_volume the image caching needs to handle
    this and treat it like a general cloning failure to failback to
    the default handling.

    Change-Id: I06ad0e71e6c502945975629141ec573110482954
    Closes-bug: #1642394

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 10.0.0.0b3 development milestone.