Cannot create a volume from a backup

[Gorka Eguileor]

Trying to create a volume from a backup fails miserably.

Command run is:

  openstack --os-volume-api-version 3.47 volume create --backup backup restored

The error in volume service is:

           |__Flow 'volume_create_manager': TypeError: restore_backup() missing 1 required positional argument: 'volume_is_new'
2023-06-28 16:18:38.158 654574 ERROR cinder.volume.manager Traceback (most recent call last):
2023-06-28 16:18:38.158 654574 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/cinder/volume/flows/manager/create_volume.py", line 1150, in _create_from_backup
2023-06-28 16:18:38.158 654574 ERROR cinder.volume.manager ret = self.driver.create_volume_from_backup(volume, backup)
2023-06-28 16:18:38.158 654574 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/cinder/volume/driver.py", line 1957, in create_volume_from_backup
2023-06-28 16:18:38.158 654574 ERROR cinder.volume.manager raise NotImplementedError()
2023-06-28 16:18:38.158 654574 ERROR cinder.volume.manager NotImplementedError
2023-06-28 16:18:38.158 654574 ERROR cinder.volume.manager
2023-06-28 16:18:38.158 654574 ERROR cinder.volume.manager During handling of the above exception, another exception occurred:
2023-06-28 16:18:38.158 654574 ERROR cinder.volume.manager
2023-06-28 16:18:38.158 654574 ERROR cinder.volume.manager Traceback (most recent call last):
2023-06-28 16:18:38.158 654574 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task
2023-06-28 16:18:38.158 654574 ERROR cinder.volume.manager result = task.execute(**arguments)
2023-06-28 16:18:38.158 654574 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/cinder/volume/flows/manager/create_volume.py", line 1241, in execute
2023-06-28 16:18:38.158 654574 ERROR cinder.volume.manager model_update, need_update_volume = self._create_from_backup(
2023-06-28 16:18:38.158 654574 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/cinder/volume/flows/manager/create_volume.py", line 1174, in _create_from_backup
2023-06-28 16:18:38.158 654574 ERROR cinder.volume.manager backuprpcapi.restore_backup(context, backup.host, backup,
2023-06-28 16:18:38.158 654574 ERROR cinder.volume.manager TypeError: restore_backup() missing 1 required positional argument: 'volume_is_new'

This happens because `_create_from_backup` method we have this call:
            backuprpcapi.restore_backup(context, backup.host, backup,
                                        volume.id)

Yet the signature of that method is:
    def restore_backup(self, ctxt, backup_host, backup, volume_id,
                       volume_is_new):

Which requires parameter `volume_is_new` to always be provided.

[Gorka Eguileor (gorka)]

And the backup is left is "restoring" state :-(

A workaround would be using the cinder backup-restore command that uses a different code path.

[Pete Zaitcev (zaitcev)]

How about this:
 https://review.opendev.org/c/openstack/cinder/+/887443

[Pete Zaitcev (zaitcev)]

The original bug, a fix for which caused this regression, is:
https://bugs.launchpad.net/cinder/+bug/2007615

This fix will need to be backported to 2023.1 once this lands on the trunk.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 22.1.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 23.0.0.0rc1 release candidate.

[Gorka Eguileor (gorka)]

Fixed in https://review.opendev.org/c/openstack/cinder/+/887443 and released

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/887851
Committed: https://opendev.org/openstack/cinder/commit/7fa491a8d9699de73b251870e7185dbe308368f2
Submitter: "Zuul (22348)"
Branch: master

commit 7fa491a8d9699de73b251870e7185dbe308368f2
Author: Eric Harney <email address hidden>
Date: Thu Jul 6 10:03:33 2023 -0400

    mypy: Add backup/rpcapi.py

    Related-Bug: #2025277
    Depends-On: Iabfebacfea44916f89584ffd019d848e53302eaf
    Change-Id: I9068b6bd32309f1c9a32c1aee57f700a4a5865b2

[zibort (zibort)]

We have a similar problem on 2023.1 (Name: cinder Version: 22.1.2.dev5). But it seems that the bug has been resolved in version 22.1.1

2024-02-01 09:42:25.916 7 ERROR oslo_messaging.rpc.server [None req-9ed72efb-e245-4a4c-80e8-53b59379ab7f 33b1d841717c47bea279d8e3fa3385fb dd1a2731ce4b493aa203487ce7e130bd - - - -] Exception during message handling: TypeError: BackupManager.restore_backup() missing 1 required positional argument: 'volume_is_new'
2024-02-01 09:42:25.916 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2024-02-01 09:42:25.916 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2024-02-01 09:42:25.916 7 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2024-02-01 09:42:25.916 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2024-02-01 09:42:25.916 7 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2024-02-01 09:42:25.916 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2024-02-01 09:42:25.916 7 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2024-02-01 09:42:25.916 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/cinder/utils.py", line 1055, in wrapper
2024-02-01 09:42:25.916 7 ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)
2024-02-01 09:42:25.916 7 ERROR oslo_messaging.rpc.server TypeError: BackupManager.restore_backup() missing 1 required positional argument: 'volume_is_new'
2024-02-01 09:42:25.916 7 ERROR oslo_messaging.rpc.server

[Gorka Eguileor (gorka)]

This sounds like the bug fix wasn't properly done, because it didn't take into consideration that we have rolling upgrade support and we can have a service send the RPC request with the old parameter count.

In the backup service we can see the rpc call checks that the RPC version being used is 2.4 and only then sends the `volume_is_new` parameter:
```
    def restore_backup(self, ctxt, backup_host, backup, volume_id,
                       volume_is_new):
        LOG.debug("restore_backup in rpcapi backup_id %s", backup.id)
        cctxt = self._get_cctxt(server=backup_host)
        if self.client.can_send_version('2.4'):
            cctxt.cast(ctxt, 'restore_backup', backup=backup,
                       volume_id=volume_id, volume_is_new=volume_is_new)
        else:
            cctxt.cast(ctxt, 'restore_backup', backup=backup,
                       volume_id=volume_id)
```

But just because the cinder-volume service hasn't started in that RPC version it doesn't mean it doesn't have the new code, because it can be running in compatibility mode as part of a rolling upgrade or because we have stale backup service entries in the DB, or because the upgrade process wasn't done right.

What we need to do is set a reasonable default for `volume_is_new` on the backup manager so that it can behave like it did before 2.4.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/914775