Error initializing IBM SVC v7000

[Gorka Eguileor]

[Bug reported by tanshaolong in old cinderlib repository https://github.com/Akrog/cinderlib/issues/15]

I am try cinderlib for IBM svc v7000, but I get below error messages. Would you help how to resolve the issue or give some hint? Thank you very much.

Test enviroment:
Cinderlib version: v0.3.9
Cinder release: Pike
Storage: IBM SVC V7000
Versions: Unknown
Connection type: FC

Test code:
import cinderlib as cl
lvm = cl.Backend(volume_driver='cinder.volume.drivers.ibm.storwize_svc.storwize_svc_fc.StorwizeSVCFCDriver',
san_ip='...',
san_login='superuser',
san_password='********',
storwize_svc_volpool_name='mdiskgrp1',
volume_backend_name='svc1234')

Error messages:
Traceback (most recent call last):
File "mytest.py", line 11, in
volume_backend_name='svc1234')
File "/usr/lib/python2.7/site-packages/cinderlib/cinderlib.py", line 86, in init
self.driver.do_setup(objects.CONTEXT)
File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/ibm/storwize_svc/storwize_svc_common.py", line 2262, in do_setup
volumes = objects.VolumeList.get_all_by_host(admin_context, self.host)
File "/usr/lib/python2.7/site-packages/cinder/objects/volume.py", line 613, in get_all_by_host
volumes = db.volume_get_all_by_host(context, host, filters)
File "/usr/lib/python2.7/site-packages/cinder/db/api.py", line 274, in volume_get_all_by_host
return IMPL.volume_get_all_by_host(context, host, filters=filters)
AttributeError: 'DB' object has no attribute 'volume_get_all_by_host'

Configuration in cinder.conf
[svc1234]
volume_driver=cinder.volume.drivers.ibm.storwize_svc.storwize_svc_fc.StorwizeSVCFCDriver
san_ip=...
san_login=superuser
san_password=*********
storwize_svc_volpool_name=mdiskgrp1
volume_backend_name=svc1234

[Gorka Eguileor (gorka)]

The issue is caused by the SVC driver accessing the database directly, which is not considered good practice in Cinder, though this code could come from a time when this was a common practice in Cinder drivers.

The problem with drivers using the DB directly is that cinderlib doesn't require a DB, it can work storing everything in memory. To facilitate storage of the resources' metadata cinderlib has its own persistence plugin mechanism, and has some ad-hoc "workarounds" for drivers that access the DB directly.

Since you are the first one to try the SVC driver, you were the one that discovered these DB dependencies on the driver which will require adding workarounds in cinderlib for the DB access that driver has.

As a temporary workaround, and to confirm that there are no additional issues with the driver, you can use one of the database persistence plugins: in-memory or using a database (it can be a sqlite database).

To use a different persistence plugin you must pass the `persistence_config` parameter to the `global_setup` method before creating any `Backend` as described in the [initialization part of the docs](https://docs.openstack.org/cinderlib/latest/topics/initialization.html#persistence-config). Database plugins are also explained in the [metadata persistence plugin doc](https://docs.openstack.org/cinderlib/latest/topics/metadata.html#database-plugin).

Example using a SQLite file called 'cl.sqlite` in the current directory:

```
import cinderlib as cl

persistence_config = {'storage': 'db', 'connection': 'sqlite:///cl.sqlite'}
cl.setup(persistence_config=persistence_config)
```

Using a memory SQLite database:

```
import cinderlib as cl

persistence_config = {'storage': 'memory_db'}
cl.setup(persistence_config=persistence_config)
```

This is a workaround because we are limited to using DB persistence plugins and we cannot use any other plugins, which means that projects like [Ember-CSI](https://ember-csi.io) will not work using their default persistence model, which in that case is storing data as CRDs in Kubernetes.

-------------------------------------------------------------------------------------

The following is only relevant for coding the fix.
Upon looking at the driver I see that we have the following DB dependencies:

- `volume_get_all_by_host` when using `objects.VolumeList.get_all_by_host`
- `volume_type_get` when using `objects.VolumeType.get_by_name_or_id`
- `volume_type_get_by_name` when using `objects.VolumeType.get_by_name_or_id`
- `volume_get` directly called by `get_by_id` when using `objects.Volume.get_by_id`
- `_group_type_get_full` directly called by `get_by_id` when using `objects.GroupType.get_by_id`

In cinderlib we don't currently support groups or consistency groups, so we don't need to implement `_group_type_get_full` and we have already implemented workaround for methods `volume_get` and `volume_type_get`, so we only need to implement workarounds for `volume_get_all_by_host` and `volume_type_get_by_name`.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/648212

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/648218

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/648212
Committed: https://git.openstack.org/cgit/openstack/cinderlib/commit/?id=385e6f4a1ec5c382885321b3db9d80f26f66c117
Submitter: Zuul
Branch: master

commit 385e6f4a1ec5c382885321b3db9d80f26f66c117
Author: Gorka Eguileor <email address hidden>
Date: Wed Mar 27 19:46:05 2019 +0100

    Add support for IBM SVC

    When initializing IBM SVC with the memory persistence we get an exception:

      return IMPL.volume_get_all_by_host(context, host, filters=filters)
      AttributeError: 'DB' object has no attribute 'volume_get_all_by_host'

    The issue is caused by the SVC driver accessing the database directly,
    which is not considered good practice in Cinder nowadays.

    The problem with drivers using the DB directly is that cinderlib doesn't
    require a DB, it can work storing everything in memory. To facilitate
    storage of the resources' metadata cinderlib has its own persistence
    plugin mechanism, and has some ad-hoc "workarounds" in the DB class for
    drivers that access the DB directly.

    The IBM driver has the following DB dependencies:

    - volume_get_all_by_host when using objects.VolumeList.get_all_by_host

    - volume_type_get when using objects.VolumeType.get_by_name_or_id

    - volume_type_get_by_name when using
      objects.VolumeType.get_by_name_or_id

    - volume_get directly called by get_by_id when using
      objects.Volume.get_by_id

    - _group_type_get_full directly called by get_by_id when using
      objects.GroupType.get_by_id

    - volume_admin_metadata_get direct call

    - volume_admin_metadata_update direct call

    - volume_admin_metadata_delete direct call

    In cinderlib we don't currently support groups or consistency groups, so
    we don't need to implement `_group_type_get_full` and we have already
    implemented workaround for methods `volume_get` and `volume_type_get`,
    so this patch implements the remaining 5 methods volume_get_all_by_host,
    volume_type_get_by_name, volume_admin_metadata_get,
    volume_admin_metadata_update, and volume_admin_metadata_delete in the DB
    class.

    Closes-Bug: #1821898
    Change-Id: I94b4864c5823976f4a9cac2e40943b74aca4066e

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/648218
Committed: https://git.openstack.org/cgit/openstack/cinderlib/commit/?id=c97367d89d341c5c2eba4102e46571ff92da2eeb
Submitter: Zuul
Branch: master

commit c97367d89d341c5c2eba4102e46571ff92da2eeb
Author: Gorka Eguileor <email address hidden>
Date: Wed Mar 27 20:16:31 2019 +0100

    Run functional tests with memory persistence

    We are currently running functional tests with the memory DB persistence
    plugin to ensure that the migrations run successfully, but this means
    that we could be hiding issues on drivers that are not 100% compatible
    with the persistence plugin mechanism, because the DB plugin still has
    the SQLAlchemy implementation behind, which other plugins don't.

    This patch first initializes cinderlib with the memory DB plugin to test
    the DB migrations, and then replaces it with the memory plugin to ensure
    driver compatibility by default.

    We can still run the tests using the Memory DB persistence model setting
    the CL_FTEST_MEMORY_PERSISTENCE environmental variable to false.

    To ensure we are constantly testing both we run the LVM gate job with
    Memory DB persistence and the Ceph gate with Memory persistence.

    Related-Bug: #1821898
    Change-Id: Id98c827ffadf2c47ff835afa8b0f83dddacb54fa

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinderlib 1.0.0 release.