cinder-volume can't be inited with large number of volume

[wangxiyuan]

the initialization of cinder-volume process has a performance issue when there a large number of volume in the backend.

With our production test, if there are one hundred thousand volumes in one backend, it will cost more than 4G memory during the process initialization. And the process will be always down because of the timeout.

We fount that the problem may be happened here:
https://github.com/openstack/cinder/blob/master/cinder/volume/manager.py#L446-L475

It firstly get all the volumes from the db, and then calculate and check them one by one.

Is it possible taht we can calculate it with the SQL command at one step to avoid memory occupancy.
Then make the "ensure_export" action be an asynchronous operation to avoid blocking the process?

[Sean McGinnis (sean-mcginnis)]

Dinesh, are you still working on this? Any ideas for a solution.

[Dinesh Bhor (dinesh-bhor)]

Hi Sean and all,

I was working on this but wasn't successful in reproducing this on my environment(which is IMO not enough for this bug). Removing myself from assignee. Please feel free to assign it to yourself.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/627634

[OpenStack Infra (hudson-openstack)]

Change abandoned by Serhii Rusin (<email address hidden>) on branch: master
Review: https://review.openstack.org/627634
Reason: false related changes

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/627636

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/629516

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean McGinnis (<email address hidden>) on branch: stable/rocky
Review: https://review.openstack.org/629516
Reason: Redo once changes are complete and merged to master.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/627636
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=ed02273d2f242a70be41f73b0909645fd9f987a9
Submitter: Zuul
Branch: master

commit ed02273d2f242a70be41f73b0909645fd9f987a9
Author: Serhii Rusin <email address hidden>
Date: Fri Dec 28 08:32:26 2018 -0500

    Initiating Cinder Volume Manager with large number of volumes

    This patch adds the option to split retrieved volumes and snapshots
    by chunks during Volume Manager host initialization. Query results
    will be obtained in batches from the database and not in one shot
    to avoid extreme memory usage. Max number of volumes and snapshots
    per batch is controlled by option init_host_max_objects_retrieval.
    Option is disabled by default for backward compatibility.

    To migrate any ConfKeyManager keys based on fixed_key to the currently
    configured key manager Volume Manager will use special lightweight
    object - VolumeMigration.

    Change-Id: I53eccc77fdc2c35b27ed430af62dc18e7d1bde69
    Closes-Bug: 1681374

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 14.0.0.0rc1 release candidate.