Horizon fails to create volume from snapshot of non-standard-type volume

[Martin Gerhard Loschwitz]

Horizon offers the opportunity to create a volume from a snapshot. It does not, however, properly determine the type of the originating volume of the snapshot and tries to use the default. In environments with non-default storage types (e.g. encrypted storages, like in our case), this fails and leads to an error message:

2020-06-26 16:13:12,633 36 WARNING horizon.exceptions Recoverable error: Invalid input received: Invalid volume_type provided: d4917a31-3332-4323-9b1f-9ace3dda9d9a (requested type is not compatible; recommend omitting the type argument). (HTTP 400) (Request-ID: req-40ee2f61-6a16-44f2-96d7-46f5c42f1942)

Problematic code is in cinder/forms.py of Horizon:

https://github.com/openstack/horizon/blob/master/openstack_dashboard/dashboards/project/volumes/forms.py#L379

Under normal circumstances, the storage type should be set in that file in "prepare_source_fields_if_snapshot_specified" (https://github.com/openstack/horizon/blob/master/openstack_dashboard/dashboards/project/volumes/forms.py#L145)

That code would need to be triggered in the __init__ function of CreateForm (https://github.com/openstack/horizon/blob/master/openstack_dashboard/dashboards/project/volumes/forms.py#L297) which tests the GET of the original request for certain strings.

Debugging however, unveils that GET simply does not contain any of the strings checked for there anymore -- in our case, it always was "<WSGIRequest: GET '/dashboard/project/volumes/create/'>". So in fact, the whole code path starting at https://github.com/openstack/horizon/blob/master/openstack_dashboard/dashboards/project/volumes/forms.py#L314 might be dead and the default will always be used. The default doesn't populate "type", however,

This of course also leads to creating volumes from volumes not working due to the same issue.

Found with the help of Harald Wagener.

[David Coronel (davecore)]

subscribed ~field-high

We see this in a bionic-train openstack deployment from the cloud archive with the following charms:

cs:openstack-dashboard-305
cs:cinder-304
cs:~openstack-charmers-next/cinder-ceph-333

With openstack-dashboard version 3:16.2.0-0ubuntu1~cloud0

There are multiple volume types in the environment, but creating a volume from a snapshot in horizon will always try to use the default type, which fails if the original volume was not created with that type.

The workaround is to use the cli:

openstack volume create <new volume name> --snapshot <snapshot> --type <type of the original volume>

[Akihiro Motoki (amotoki)]

Cinder picks up an appropriate volume type based on a snapshot information when volume_type is not specified, so horizon does not specify a volume type when creating a volume from a snapshot.

At least I confirmed that horizon stable/train works with cinder master (almost same as stable/victoria; this is due to my dev env). I created a second volume type that is not the default volume type and it works.

I wonder how cinder behaves.
What happens when you create a new volume from a snapshot in CLI WITHOUT specifying a volume type?

Cinder picks up an appropriate volume type based on a snapshot information when volume_type is not specified.
As of Victoria (my dev env), a volume type is picked up from a source volume of the specified volume snapshot expectedly [1]. (horizon stable/train also works well)

[1] CLI log http://paste.openstack.org/show/799064/

[Akihiro Motoki (amotoki)]

Note that I tried two cases.
(1) Project -> Volumes -> Volumes -> Create Volume, Select "Snapshot" as "Volume Source" and choose a snapshot with a second volume type
(2) Project -> Volumes -> Snapshots -> Create Volume (which populates a target volume snapshot by default)

[David Coronel (davecore)]

The same steps give me a different outcome. I guess I must be running an older version of cinder:

$ openstack volume type list
+--------------------------------------+--------------------+-----------+
| ID | Name | Is Public |
+--------------------------------------+--------------------+-----------+
| 17593eb4-a4d2-4464-82ea-af22c49f4105 | stable1-az2 | True |
| 3421912b-0ebc-4c53-b318-a95edfe3eb86 | arbor-az1 | True |
+--------------------------------------+--------------------+-----------+

$ openstack volume type list --default
+--------------------------------------+-----------+-----------+
| ID | Name | Is Public |
+--------------------------------------+-----------+-----------+
| 3421912b-0ebc-4c53-b318-a95edfe3eb86 | arbor-az1 | True |
+--------------------------------------+-----------+-----------+

$ openstack volume create --type stable1-az2 --size 1 david-stable1-az2-volume1
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| attachments | [] |
| availability_zone | zone2 |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2020-10-16T18:38:05.000000 |
| description | None |
| encrypted | False |
| id | 7b1f4f0b-61d5-44c5-8c47-8828de477afc |
| migration_status | None |
| multiattach | False |
| name | david-stable1-az2-volume1 |
| properties | |
| replication_status | None |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| type | stable1-az2 |
| updated_at | None |
| user_id | f2ee9e8060e54d058060d220fab84088 |
+---------------------+--------------------------------------+

$ openstack volume list | grep david
| 7b1f4f0b-61d5-44c5-8c47-8828de477afc | david-stable1-az2-volume1 | available | 1 | |

$ openstack volume snapshot create --volume david-stable1-az2-volume1 david-stable1-az2-volume1-snap1
+-------------+--------------------------------------+
| Field | Value |
+-------------+--------------------------------------+
| created_at | 2020-10-16T18:38:40.904359 |
| description | None |
| id | 23dfa183-1290-4c76-9aae-a35f53d94c6e |
| name | david-stable1-az2-volume1-snap1 |
| properties |...

[David Coronel (davecore)]

I also tried with an original volume with a type in the same AZ, but I have the same issue:

$ openstack volume create --type stable1-az1 --size 1 david-stable1-az1-volume1
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
[...]
| availability_zone | zone1 |
[...]

$ openstack volume create --snapshot david-stable1-az1-volume1-snap1 david-stable1-az1-volume1-snap1-volume1

Invalid input received: Invalid volume_type provided: 3421912b-0ebc-4c53-b318-a95edfe3eb86 (requested type is not compatible; recommend omitting the type argument). (HTTP 400) (Request-ID: req-e26f466a-79da-4fbd-9f9d-785a0ea2b60f)

[David Coronel (davecore)]

I notice that my _get_volume_type volume creation flow code is different from the one in master[1]:

====================
    def _get_volume_type(self, context, volume_type,
                         source_volume, snapshot, image_volume_type_id):
        if volume_type:
            return volume_type
        identifier = collections.defaultdict(str)
        try:
            if source_volume:
                identifier = {'source': 'volume',
                              'id': source_volume['volume_type_id']}
            elif snapshot:
                identifier = {'source': 'snapshot',
                              'id': snapshot['volume_type_id']}
            elif image_volume_type_id:
                identifier = {'source': 'image',
                              'id': image_volume_type_id}
            elif CONF.default_volume_type:
                identifier = {'source': 'default volume type config',
                              'id': CONF.default_volume_type}
            if identifier:
                return objects.VolumeType.get_by_name_or_id(
                    context, identifier['id'])
        except (exception.VolumeTypeNotFound,
                exception.VolumeTypeNotFoundByName,
                exception.InvalidVolumeType):
            LOG.exception("Failed to find volume type from "
                          "source %(source)s, identifier %(id)s", identifier)
        return None
====================

[1] https://github.com/openstack/cinder/blob/master/cinder/volume/flows/api/create_volume.py#L352

[David Coronel (davecore)]

I found this cinder bug which is fix released:

Can't restore volume to a different type than the snapshot
https://bugs.launchpad.net/cinder/+bug/1289931

[David Coronel (davecore)]

That bug mentions:

    This patch allows the capability, but checks validity at the
    API layer before issuing the create call. There are two
    requirements for the new type specification to be valid:
    1. There is only one backend (cinder-volume) topic configured
    2. Both types in question specify the same volume_backend_name
    If neither of these requirements are met, the user will receive
    an "invalid type" error explaining that the type combination is
    not compatible and that they should omit the type argument altogether.

    Change-Id: I08bc5e9a8800ce3b27c7db90b7bff86d7d14359a
    Closes-Bug: #1289931

In my environment, we have 6 enabled_backends (2 AZs with 3 backends in each one).

@Akihiro: Do you have just one enabled_backends in your cinder.conf?

[Corey Bryant (corey.bryant)]

@David, have you tested on ussuri or victoria by any chance?

[Corey Bryant (corey.bryant)]

This may be fixed in https://bugs.launchpad.net/cinder/+bug/1879578

[David Coronel (davecore)]

@Corey, no I only have access to this bionic-train cloud.

[Corey Bryant (corey.bryant)]

1879578 is fixed in cinder 15.4.0 for train and 16.2.0 for ussuri. These are new releases that we haven't picked up yet in Ubuntu.

[Corey Bryant (corey.bryant)]

cinder 15.4.0 and 16.2.0 will be included in the following SRUs for Ubuntu:
https:pad.lv/1900477
https:pad.lv/1900476

[Corey Bryant (corey.bryant)]

I'm going to mark this as a dup of 1879578. If that is not correct, please let me know.